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ABSTRACT 

The Massachusetts Workplace Education Initiative 
(MWEI) was begun in 1985 as an inter-agency effort to bring adult 
basic education and English-as-a-Second-Language (ESL) instruction 
directly into workplaces throughout the state. The impetus for the 
program was a concern for large numbers of workers who did not have 
the skills necessary to compete successfully for n^w jobs after plant 
closings or other layoffs. This report details what came to be known 
as the "outcome study" and the "curriculum study" of the MWEI Cycle 4 
Evaluation, as well as the "termination study," which developed when 
one company closed and another stopped its education program midway 
through the outcome study. The overall evaluation goals were: (1) to 
develop a process for defining and measuring meaningful outcomes, and 
(2) to assess the organization, delivery, and effectiveness of 
educational materials and curricula in four workplace education 
programs. Information on the following six sites are described in the 
outcome and curriculum studies: Hampden Paper Company (Holyoke) ; 
Aerovox, inc. (New Bedford) ; Kennedy Die Castings (Worcester); united 
Electric Controls Company (Watertown) ; South Cove Manor Nursing Home 
(Boston); and Boston City Hospital (Boston), in addition, case 
studies are reported for Friction Materials, inc. (Lawrence) and 
Spir-it, Inc. (Wakefield) in the termination study. Recommendations 
are offered for future evaluation and technical assistance. Nearly 
half the document consists of outcome study case studies, which are 
appended; also appended are numerous questionnaires used in the 
study. Contains 10 references. (LB) Adjunct ERIC Clearinghouse on 
Literacy Education. 
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"Deep concern over America's declining economic role in the world might 
provide an opportunity to emphasize the centraHty of education to national 
objectives. But this would require an end to the widespread reluctance on 
the part of firms to invest more substantially in training and to reorganize 
the workplace in ways that would promote continuous learning." 

Made in America. Commission on Industrial Productivity, the Massachusetts 
Institute of Technology, Cambridge, MA, 1989. 



"In today's workplaces, change is often unexpected, unavoidable and 
threatening...Worklng together as equals, unions and employers have been 
able to plan for change and reduce or eliminate Its harmful effects. A 
worker education strategy mat includes workplace literacy is frequently an 
Important part of that agenda for change." 

"Worker-Centered Learning: A Union Guide to Workplace Literacy", Anthony 
Sarmiento and Ann Kay, AFL-CIO Human Resources Development Institute. 
Washington, D.C.:1990. 



"Changes In the economy imply that the need to know how to learn - in 
other words, how to organize social and technological resources, to 
transform what Is familiar into the mastered- is a process that requires 
knowing how to identify the limits of one's own knowledge, how to ask 
germane questions, how to penetrate poor documentation, and how to 
identify sources of information." 

Sue E. Berryman: "Education and the Economy: A Diagnostic Review and 
Implications for the Federal Role." 



"increased self-confidence seems to be the most Important outcome of the 
workplace education program because It correlates with so much, if a 
person is self confident, he'll learn more, communicate better, and do better 
work." 

Supervisor at the TJMAXX Distribution Center, Worcester, MA. 

^ "Being in this program has improved every part of my life. Now I like to 
% learn. You can't stop me! This means I'm doing better at work and I even 
^ read to my kids at night-." 

O Employee, the Belmont Home, Worcester, MA. 
C 
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Executive Summary 



Introduction 

In November of 1990, the Bureau of Adult Education in the Massachusetts 
Department of Education contracted with Evaluation Research, an independent 
consulting company, to conduct an evaluation of the fourth cycle of operation of 
the Massachusetts Workplace Education Initiative (MWEI). The broad goals of 
the MWEI Cycle 4 evaluation were: 

1 . to develop a process for defining and measuring meaningful "outcomes" in 
eight workplace education programs, a process which would become the basis 
for an ongoing, customized evaluation system in each program; 

2. to review the organization and delivery of educational materials and curricula 
in four workplace education programs and to assess the overall effectiveness 
of these materials and curricula. 

This report describes the rationale for, and the methods and results of what came 
to be known as the "outcome study" and the "curriculum study" of the MWEI Cycle 
4 Evaluation. Midway through the outcome study, one company closed down 
and another stopped its education program. As a result, a "termination study" 
became the third part of the MWEI Cycle 4 Evaluation. 

Background of this Evaluation 

The Massachusetts Workplace Education Initiative was begun in 1985 as an 
inter-agency effort to bring adult basic education and English as a Second 
Language (ESL) instruction directly into workplaces throughout the state. State 
policy makers had at that time become concerned about the large numbers of 
workers who did not have the basic skills to apply successfully for new jobs after 
plant closings and other lay-offs. This concern led the former Office of Training 
and Employment Policy in the Executive Office of Economic Affairs, the Bureau of 
Adult Education in the Department of Education, the Executive Office of Labor 
and, later, the Commonwealth Literacy Campaign to form a partnership to 
upgrade workers' skills before they lost their jobs, ft was felt that a more-skilled 
workforce would strengthen business, enhance competitiveness in a harsh 
economy, and save jobs. Using state and federal funds, the MWEI partnership 
has provided state (and, in some cases, federal) funding and technical assistance 
to create local workplace basic skills education programs around the state. Since 
the mid-1980s, the MWEI partnership has commissioned a series of evaluations 
which document the varied workplace education service delivery models being 
developed. 



The Outcome Study 

Goals: 

Building on the findings of prior evaluations, the MWEI in Cycle 4 asked the 
evaluators to develop a process which workplace education programs can use to 
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assess their outcomes in terms that are more specifically relevant to bmployers', 
unions*, and learners* goals. This process was designed so that any workplace 
program could adapt it to its own evaluation needs and resources and develop its 
own ongoing, customized evaluation system. Thus, this was not a traditional 
"outcome study" conducted by outsiders but rather an effort combining 
organizational development and outcome evaluation activities. 

The goals of the outcome study were to: 

1. Build the capacities of individual programs to conduct outcome evaluations. 

2. More clearly define desired outcomes for each site and produce data which 
demonstrate what in fact is being achieved. 

3. Develop a "tool kit" (an evaluation planning process, data gathering 
instruments, and resource persons) which other programs can use to 
develop their own planning and evaluation systems. 

The ten-step process used in the six sites: 

In each of the participating programs, the consultants established "evaluation 
teams" which were made up of representatives from all areas of the program 
partnership - upper-level managers, supervisors, plant managers, human 
resource development officers, workers enrolled in the program, union 
representatives, and programs coordinators and teachers. Over the course of the 
year, following a ten step process, the evaluation team members identified the 
outcomes they believe to be important to their programs and companies; 
designed instruments to best measure whether these outcomes were in fact 
being achieved; collected, analyzed, and reported the data; and also evaluated 
the process which they had employed to measure their outcomes - in the service 
of improving that process next year. 

The following is a description of the ten-step process which was employed at the 
Hampden Paper Company (Holyoke); Acruvox, Inc. (New Bedford); Kennedy Die 
Castings, Inc (Worcester); United Electric Controls Company (Watertown); South 
Cove Manor Nursing Home (Boston); and Boston City Hospital (Boston,) as well 
as a general assessment of how successful the process was. 1 

?teP 1 '■ Evaluators familiarize themselves with tl e programs: December 
1990 - February, 1991 

The consultants familiarized themselves with the programs through review of 
documents from previous evaluations, interviews with program coordinators and 
on-site visits at the workplaces themselves (which were concurrent with the first 
on-site evaluation team meetings). 



llSlote that the data-gathering instruments used and the findings at each site are described in 
detail in the case studies contained in Appendix C, under separate cover, as well as in the 
complete report ot the Cycle 4 MWEI Evaluation. 
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Step 2: Program coordinators assemble their evaluation teams: December - 
January, 1990 

At each site, program coordinators identified the people who would best 
represent the workplace education program partnership on the evaluation team, 
and brought them together for an orientation meeting. 

Step 3: Teams from six sites meet as a group: January, 1990 

Representatives from ail six sites met in Boston where they got to better know 
their fellow team members, and where they were introduced to people 
representing similar interest groups in the other workplace education programs 
from around the state. The teams were challenged to work in various kinds of 
group activities to begin clarifying for their respective sites answers to this 
fundamental outcome evaluation question: "Who wants to know what information 
for what purpose?" More specifically, this question was broken down into sub- 
questions: (1) Who are the interest groups represented in your program? (2) What 
information do those various interest groups want to develop in this evaluation? 
(3) Why do they want to know that information? 

Step 4: Each team further defines "Who Wants to Know What Information for 
What Purpose and How Will that Information Be Gathered?": February - 
March, 1991 

The consultants helped the teams to further define what information they wanted 
to get in their study and how they would get it. They accomplished this at the first 
of three or four site visits to the sites, during which they conducted their first on- 
site "evaluation team meeting." 

Step 5: Teams agree on details of the data-gathering activities to be used: 
April - May, 1991 

The consultants helped their teams finalize details of how data would be 
gathered. They did this at the second team meeting, and in thw preparations for it. 
The program coordinators generally took the lead in their teams to finalize the 
answers to the following questions: Which data-gathering activities will we use? 
For each data-gathering activity, what will be the content and format? How will 
the data be used? How will the data be collected? How will the data be 
analyzed and reported? What resources need to be committed and under what 
time-line? 

StepS: Teams gather data: May -September, 1991 

The actual gathering of data in the participating programs went more or less 
according to schedule. However, some of this data-collection was complicated 
by unforeseen staff changes and interruptions of work schedules. It is inevitable 
that unforeseen problems will surface. The key to continuing a successful data 
gathering process is flexibility. 
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Step 7: Special training provided on data analysis: June, 1991 



In response to a request from some coordinators for assistance with 
computerized analysis of the data, the consultants arranged a half-day training 
course for any coordinators interested in learning how to use the Statistical 
Program for the Social Sciences (SPSS) computer software, it was discovered 
that the steps involved in using the program also help teams to clarify what the 
desired program outcomes are. 

Step 8: Data analyzed: June - November, 1991 

From June through October, the teams tabulated and analyzed the de*a they had 
gathered in Step 6. This was done in part with input from the two consultants. 
The consultants first reviewed the data which had been gathered. They tnen 
gave feedback to the teams via telephone conversations with program 
coordinators and in the third and fourtn on-site meetings which the consultants 
held with the evaluation teams. 

Step 9: Reporting of data and other follow-up action: August - October, 1991 

Having summarized and analyzed the data, the six teams were faced with the 
task of preparing reports for the audiences they wished to present their findings 
to. The teams decided to whom they would report their findings and the formats 
of those reports. This was a very important step because it resolved the question 
of how the data would be used. Oi tcome evaluation would not be an idle 
exercise, but a real effort to document program success for potential funders and 
other audiences. 

Step 10: Evaluation done of the outcome study process itself: September - 
November, 1991 

In order to clarify how effective this outcome study process was, the consultants 
provided opportunities for the six evaluation teams to assess their own 
experience in the project and to make recommendations for possible future 
outcome study activities. 



Conclusions: 



» The ten-step outcome study process takes th e m wei far closer this y ^r t n 
demonstrating a "convincing" link between basi c skills ipstnjcjifla t hfl 
workplace and improved iob performance. The task of developing meaningful 
measures of a "convincing link" between basic skills instruction at the workplace 
and improved job performance is a difficult one which requires real, ongoing 
collaboration among the partners represented in a workplace education 
program: employers, unions, educators, and workers themselves. One 
effective vehicle for building this collaboration is an "evaluation team" in which 
these interest groups together identify the outcomes which they hope their 
education programs will achieve, measure what outcomes are in fact being 
achieved, and demonstrate the link between the education program and 



increased productivity and other desired outcomes. The evaluation team has 
also became a vehicle through which the information gathered in an evaluation 
can be used to improve program services. 

« The limitations of traditi onal data-gathering methods can be overcome in 
subsequent studios Teams typically decided to produce questionnaires, 
surveys and checklists, which would permit quick, non-obtrusive data gathering. 
With hindsight this reliance on "tried and true" data-gathering methods limited 
the results of the study. There was, in the end, an abundance of learner self- 
report data, and somewhat less data submitted by supervisors - who were so 
protective of their time that tt<ey often responded to questions about changes in 
the worker/learners in the aggregate. That is , they did not fill out a 
questionnaire for each learner in their department. Rather, they filled out one 
form which described their perception of the overall change in all the learners in 
their department. This was probably an inevitable limitation of the first year "try- 
out" study, because the team members in general could not commit the time to 
anything more than checklists and other easy-to-use instruments. In 
subsequent evaluations, however, teams should be better prepared (with the 
full support of upper management) to commit the time required for thoughtful 
evaluation. Teams should also be given the technical assistance they need to 
use more-creative and meaningful data-gathering instruments. These 
preparations will result in more creative and accurate data. 

* Outcome evaluations should define "outcomes* bro adly, not narrowly . All the 
evaluation teams were interested in identifying and measuring outcomes which 
would demonstrate the link between their education programs and specific 
improvements in employee job performance. These outcomes, in general, 
consisted of improved abilities to apply reading, writing, oral communication, 
and math in specific job functions. However, all the teams were also interested 
in broader outcomes which represent a new order of skills demanded in the 
changing Massachusetts (and American) workplace: for example, problem- 
solving, taking initiative, and working effectively with a team. Many of the 
evaluation teams were also interested in outcomes that focus on students' 
personal development (for example, going on to college, being able to 
communicate with neighbors), on their understanding of their rights as workers 
and citizens, and on their participation in their unions. 

All of these skills were seen as complementary and integral to the "whole" 
learning which makes adapting to new challenges in the workplace possible. 
Thus, the pool of outcomes identified by the teams reflected an understanding 
of how an employee's productivity can be strengthened by helping the worker to 
develop a mix of skills at the same time; immediately-job-specific skills, broader 
skills transferable across a number of jobs, and "life skills" which not only 
enhance the employee's personal development but are transferable back to the 
job. 

* Outcomes vary widely from context to context, iob to job, worker to worker . 
Some examples of reported outcomes include: 



- Reading production tick ets accurately, a critical part of the production 
process at Hampden Paper, has improved. Sixty percent of the 
supervisors who responded to the supervisor questionnaire noted 
improvement in workers' accuracy in reading production tickets. Eighty- 
seven percent of the students who responded to the student questionnaire 
noted improvement in reading production tickets accurately. 

- The unanticipated improvement in workers' safety-related hehavinr at 
Aerovox may save the company a significant amount of money by reducing 
the amount of worker's compensation insurance which the company must 
buy. (Over 81% of the workers noted improvement in taking safety 
precautions, as did 66.7% of the supervisors. 

- The education program at South Cove Manor is credited with 
significantly increasing positive interaction between residents and nursing 
assistants. For example, residents feei they now interact positively with 
91.6% of students. 

- At Kennedy Die castings, the supervisor who twice evaluated the 
students noted that students, on average, improved slightly during the 
period of June to A«jgus>l in their abilities to handle all of the given job 
tasks, and that they showed the strongest abilities in identifying quality 
dejects, functioning as a company employee, and comprehending and 
following directions. 



The Curriculum Study 

The curriculum study consisted of a careful review of curriculum development at 
four MWEI programs. Its purpose was to evaluate curriculum, which was defined 
as the materials developed and used by program staff, the process by which 
materials are put together, actual classroom practices, and the philosophy which 
guides these activities." Each curriculum was assessed against a set of 
standards or norms synthesized from major adult basic education documents 
which represent the MWEI partners' best thinking on program effectiveness. 

The study began with a review of critical documents in order to develop the set of 
standards. From this set of standards, protocols were created for document 
reviews, classroom observations, and interviews. Each site was visited for at 
least two days, and student/workers, teachers, coordinators, and evaluation team 
members (representing management, labor and the education provider) were 
interviewed. After the information was analyzed, a draft was returned to the 
programs for a review for accuracy, and for additional information. 

Findings are reported for each of the individual programs and are compared 
across the four programs. Recommendations are made for individual programs, 
as well as for the state Initiative. 



The study concludes that there are some apparent trends or patterns across the 
four MWEI programs: 

1 . Curriculum is holistic, participatory, created or tailored at each site, and 
addresses needs of student/workers, companies, and organized labor; 

2. Teachers are critical to program success, and the role requires significant 
experience and talent; 

3. Instructional methods and materials are varied in all the programs; but one 
important mode of learning, computer-assisted instruction, is lacking in all four 
programs; 

4. Student/workers at all four programs participate in curriculum decision-making; 

5. The curricula at all four programs are work-related, but also include other skills 
and knowledge which student/workers need at work and in other parts of their 
lives; 

6. The programs' curricula are meeting MWEI standards; 

7. Across the state Initiative, how to meet the full range of student/workers' basic 
skills needs through these small programs is an unsolved problem; 

8. The curricula have an impact on workplace culture; and 

9. There are site-produced curriculum products and processes which could be 
refined and disseminated. 



Termination Study 

The termination study was bom midway through the evaluation year when one 
company- Friction Materials, Inc. (FMI}~ ceased operation and another, Spir-it, 
Inc. (Spir-it) became unable to sustain its education program. This study: 

(1) describes and analyses how the goals of the FMI ESL program were linked to 
company growth and profitability and examines why the program - and the 
company - could not fulfill its goals; 

(2) examines why Spirit, Inc. was no longer able to sustain its education program, 
and addresses in particular what role basic skills education played in the 
company's long term strategy to bring about needed organizational change. 

The evaluator conducted a series of interviews with program staff and company 
representatives at each site and then wrote case studies on each program. The 
evidence and analysis presented here are intended to guide discussion about 
workplace education in ways which link its success or failure to broad company, 
union, and worker goals. 
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Friction Materials, Inc., Lawrence 



FMI was a small (100 to 300 employees}, fifty-year-old manufacturer of brake 
technology. Five years ago, the company was on the brink of bankruptcy, in part 
due to heavy investments in new, non-asbestos based materials, and in part 
because it was losing customers from a worsening reputation for poor quality. 
The once successful company at that point negotiated a transfer of assets to 
Ecklin, Inc.- a large and successful manufacturer of automotive parts. By 
February 1989, when the FMI on-site ESL program was begun, FMI had grown 
from 1 00 to 260 employees. 

The ESL program hoped to address some of the problems the company still 
faced despite its growth: (1) poor retention of workers; (2) poor communication 
between English-speaking supervisors and Spanish-speaking workers; (3) 
inadequate skills which prevented many workers from participating in a company- 
wide program in statistical process control; and (4) financial loss from excessive 
waste in the production process, also to be addressed by the new SPC system. 

In late winter/early spring 1989, with enrollment high, it seemed that the FMI 
program was one of the best examples of a successful workplace education 
program in the MWEI. But, by late summer, over 100 workers had been laid off 
and classes were reduced by over two-thirds. The reason given for the lay-offs 
was lack of orders caused by a small recession. Much effort went into keeping 
the ESL program alive during this time. Clear communication between workers 
and supervisors was critical oecause (1) Ecklin began production of a more- 
compjcated brake which required workers to take instructions from supervisors- 
and (2) a company-wide SPC program was being instituted that demanded good 
English-language skills. However, Ecklin announced that it would shut down 
operations at FMI a week before the teacher was to pilot a new quality-awareness 
program. Ecklin cited fiscal reasons for closing the plant. Did the ESL program 
live up to rts promise? If so, was the program simply not enough to help save the 
company? Or was the program never fully realized as part of a genuine strategy 
for company change? If it was never fully realized, why wasn't it- given the 
apparent agreement on goals among the key players? The information collected 
dunng the termination study attempted to answer these questions. 

Conclusions 

• Despite the agreement on goals among the key players, the program did not live 
up to its promise. There was not enough support from upper management to 
develop it fully or to carry it through hard times. 

• Under pressure to meet quotas with a diminished workforce, management 
seemed to pit immediate production goals against a long-term commitment to 
education and organizational change. But this is more complicated in an 
industry where there is no history (and no need until now) of understanding the 
relationship between education and improvement. 

• Management's commitment to instituting a complete quality control program was 
questionable. There was even a question about the extent to which the quality 
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program was intended to be an appeasement for one large customer as 
opposed to a method which genuinely guided production and standards for ail 
customers. 

• Because of the muddied messages being sent by management about education 
and quality, the workers were not motivated to attend the ESL. class. 

Spir-it, Inc., Wakefield 

Spir-it , Inc. is a 35-year-old company of about 125 people which produces plastic 
stir-sticks, straws, picks, spears, and food markers. Purchased from the original 
owner in 1988 by an investment group, the company hoped to grow over the next 
two years to annual sales of $15 million. This new management plan, coupled 
with the need to automate production to meet changing customer demands, 
underscored the need for on-site instruction to workers whose basic skills and 
English-language proficiency were limited. 

In the spring of 1989, an ESL program was begun for a small group of workers. 
Workers attended class on their own time after the main shift. After its first year, 
the program seemed successful enough but then enrollment dropped to a 
handful. When Spir-it showed a negligible profit in the second program yen;, 
management decided to make the changes which they had been talking about for 
some time: (1) consolidate production functions; (2) automate; and (3) institute 
SPC. The Spir-it education advisory board tried to answer: How can the 
educational services offered through the MWEI most help the company in a time 
of transition? The answer included a plan to offer a pre-SPC course on company 
time for all employees that would raise consciousness about quality and prepare 
the way for SPC. 

The coordinator developed the pre-SPC curriculum but, shortly before it was to 
be piloted, the program foundered on the issue of release time for line workers. 
Management decided it could not offer any program to workers during shift time 
as planned because of increasing pressure to meet production quotas. 
Management decided to offer the pre-SPC program to supervisors and managers 
who, as salaried workers, could be asked to put the extra time into training 
without pay. These people would then, in turn, teach the set-up people, line 
workers and operators during their shifts. This solution did not meet with the 
standards set by the state funder for proper use of workplace education funds for 
basic skills education. When management further indicated to the state 
coordinator that no enticement could be made to line workers to contribute their 
own time to the education program because they might be laid off in the near 
future, a fundamental problem became even clearer: The target group defined by 
federal guidelines would probably no longer be employed. 

Conclusions 

• Although it appeared that Spir-it was developing a company-wide educational 
strategy to increase production in the long run, the company did not make a real 
investment in education when production was tight. The management at Spir-it 



seems to have felt so desperate in the face of continually falling profits that they 
could not entertain an alternative perspective, one which may have helped them 
improve production through an investment in education. 



Recommendations 
Outcome Study 

The following summary recommendations for the outcome study are divided into 
two sets. The first set is for state-level planners and funders; the second is for the 
evaluation resource people who would provide technical assistance to evaluation 
teams in the future. 

In order to build on the strengths of the first year's experience and avoid the 
problems encountered, state-level planners and funders of workplace education 
programs should: 

» Modify funding guidelines to require that each site have an active planning and 
evaluation team in nlace if it is to qualify for funding. These teams would be 
more than figurehead groups. They would be required to work together over 
the course of the funding period to perform appropriate planning and evaluation 
activities. They would need an active coordinator in charge of organizing 
planning and evaluation activities. These activities would include ongoing 
goal-setting activities and development and use of appropriate measures of 
program outcomes. Program coordinators are the likely people to take on this 
role. 

* Require these planning and evaluation teams to follow a ster>b y -stap p m^** 
similar to that developed in this first year's nmiflcf With access to a core group 
of resource persons having technical expertise in the various facets of planning 
and evaluation work, the evaluation teams could efficiently incorporate planning 
and evaluation as a central function of their work, as they did in Cycle 4. 
Already-existing teams would continue to receive technical assistance from 
resource persons. The state-level partners would also organize this core group 
of resource persons in Cycle 5. 

' Provide additional workshops where supervisors, man a gers, workers union 
representatives, and education staff me et together in their interest grn i m s 
across programs. The outcome evaluation this year only touched on the benefit 
of this kind of cross-fertilization in the January 10th workshop. With the 
exception of education staff who already enjoy regular meetings with their 
counterparts from other programs, all of the participants at that workshop said 
that they wanted more time with other people around the state who share their 
particular interests. 

In addition to what the state-level partners need to do, a good deal of activity is 
needed at the level of the local workplace education partnerships. If the 
partnerships are re-organized under "planning and evaluation teams," then the 
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members of those teams must be educated about the value and technical 
requirements of effective program evaluation. Specifically, to best facilitate team 
based evaluations in the second year, evaluation resource persons should: 

• Provide training opportunities in which teams can develop a team identity and 
negotiate goals to be achieved through collaborative program evaluation. 

• Develop additional conceptual frameworks and group exercises through which 
teams can clearly define desired workplace education outcomes. 

• Actively address the contradiction between programs' desires to measure 
meaningful outcomes and the limited amount of time they feel they can invest in 
collective planning, data gathering, and analysis. 

• Explore different ways to collect data beyond the checklist, survey, and 
interview; develop a "menu" of methods for use across programs, including 
both "direct" and "unobtrusive" methods. 

• Consult with a specialist in measurement to ensure that evaluation tools are 
methodologically correct. 

• Gather the teams together for more frequent sharing of information. 

• Expand the computer data-analysis expertise among teams for the following 
reasons. Computer analysis can: 

- help to systematize the sequence of steps in conducting an outcome study; 

- establish a system at the beginning of a study - for even a small group of 
respondents - which can be expanded and compared over time; 

-« over time help to bring other data-collection procedures in the workplace into 
a comparable format; 

- present data in a statistical format which appeals to managers and others who 
want "hard" information about how the program is working; 

- provide another way for students to become involved in their programs and 
their evaluations. 

A second year of planning and evaluation activity could thus combine guidelines, 
funding, and technical assistance to create a new orientation and new capacities 
in ail workplace education programs in the state. This second year's activity 
should itself be evaluated and in turn built on in future years. 

Curriculum Study 

The Massachusetts Workplace Education Initiative has wisely encouraged each 
of the programs to develop its own curriculum, tailored to meet its own local 
needs. However, at this point in the development of the Initiative, we believe that 
the programs could benefit from a more systematic approach. We recommend 
the following actions and indicate in parentheses who might take responsibility 
for them: 7 
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1 . Expand the use of evaluation team to all current and new programs, to enable 
programs to define and measure program outcomes and to evaluate 
programs* diagnostic/placement and student progress assessment systems 
{Evaluation Team). 

2. Provide an opportunity for programs to share, publish, and compare their 
intended outcomes (State Level Partners). 

3. Ask advisory committees and those responsible for curriculum development at 
each site to review their curriculum and classroom assessment instalments in 
terms of newly-clarified intended outcomes, and to refine (add and eliminate) 
curriculum units and instructional activities to increase achievement of these 
agreed-upon goals (Program Sites). 

4. Examine intended outcomes across the initiative to determine which of these 
goals are widely shared, and publish a document listing these. Ask individual 
programs to list commercial and home-grown curriculum materials which they 
have found effective in helping student/workers achieve these outcomes; and 
prepare, publish and distribute a matrix of these materials indexed to their 
intended outcomes. This will serve three purposes: (1) it will provide a useful 
self-evaluation activity for each program, as it sees for which goals it has or 
doesn't have adequate instructional materials; (2) it will provide a useful way 
for programs to share materials across the Initiative; and (3) it will provide a 
clear picture of where materials need to be developed across the Initiative 
(State Level Partners, through the System for Adult Basic Education Support 
[SABES]). 

5. Using the set of draft curriculum standards which we prepared for this study (or 
another draft set of curriculum standards) ask representatives from all 
programs in the Initiative to review, critique, and accept a set of initiative-wide 
curriculum standards. Use these standards in requests for proposals, and use 
them for periodic peer evaluation of workplace education programs (State 
Level Partners and Program Sites). 

6. Staff development is important for all workplace education teachers 
and other staff. Each staff member should have an annual staff 
development plan which is related to areas of the job in which this 
person agrees that further knowledge, skills, or practice are needed 
(State Level Partners and Program Sites). 

7. As workplace education teachers need to share their problems, solutions, and 
expertise, we recommend that regional workplace education teacher-sharing 
networks be created in at least two other parts of the state, the central and 
western regions. They could be modelled after the successful existing 
network in the Boston area sponsored by the ALRI/SABES Greater Boston 
Regional Center. These networks could use the above matrix as a tool for 
teachers sharing curriculum materials and for collaboration to develop new 
materials across programs (State Level Partners, through SABES). 
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8. Copies of workplace education curricula and curriculum materials developed 
locally and elsewhere should continue to be collected at the ALRI/Greater 
Boston SABES Regional Center library and at the State Workplace Education 
Library, and lists of these materials should continue to be updated and 
circulated among workplace education programs (SABES). 

9. The orientation for adult basic education staff developed by Quinsigamond 
Community College for its adult education teachers should be reviewed by 
SABES as a possible model for orienting new workplace education teachers 
regionally across the state (SABES). 

10. The position of state Workplace Education Coordinator should be restored in 
order to coordinate these and other state-level responsibilities (State Level 
Partners). 

1 1. The workplace education programs should consider the advantages of 
computer-assisted instruction, and purchase computers. The state staff- 
development and support system should provide training to workplace 
education teachers in how to use computers for instructional purposes and as 
tools for word processing, and information storage and retrieval (State Level 
Partners, Program Sites, and SABES). 

12. As in many adult basic education efforts, the systematic teaching of 
mathematics is weak or non-existent in these programs. Mathematics 
instruction, possibly using computer-assisted instructional software, should be 
made a higher priority, particularly in programs where mathematics 
knowledge and skills are required to perform work tasks (Program Sites). 

13. If the use of computers in workplace education programs in small-to medium- 
sized companies is to become a reality, resources for computers and 
computer-assisted instruction need to be made available to programs through 
both public and private sources. In particular, we recommend that the Federal 
Workplace Education Initiative address this problem in its guidelines and 
requests for proposals. 



Termination Study 

• The MWEI should encourage employers and unions to think about education as 
a long-term strategy for improving workers' skills and reorganizing the 
workplace. This encouragement may take the form of a workshop, seminar, or 
panel presentation designed for the business community and organized labor. 

• The MWEI should solicit the public support of business and union leaders who 
are invested in education as a long-term strategy for change in the workplace, 
and involve them in workshops and other presentations on the topic. 
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The MWEI should consider more deeply than it has already how ethnicity and 
class influence program success. It should include these issues in discussions 
and workshops about workplace education. 

The MWEI should consider first-language instruction in those settings where it is 
an appropriate strategy. 
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Introduction 



In November of 1990, the Bureau of Adult Education in the Massachusetts 
Department of Education contracted with Evaluation Research, an independent 
consulting company, to conduct an evaluation of the fourth Cycle of operation of 
the Massachusetts Workplace Education Initiative (MWEI). The broad goals of 
the MWEI Cycle 4 evaluation were: 

1. to review the organization and delivery of educational materials and 
curricula in four workplace education programs and to assess the overall 
effectiveness of these materials and curricula; 

2. to develop a process for defining and measuring meaningful 
"outcomes" in eight workplace education programs, a process which 
would become the basis for an ongoing, customized evaluation system in 
each program. 

This report describes the rationale for, and the methods and results of what 
came to be known as the "curriculum study" and "outcome study" of the MWEI 
Cycle 4 Evaluation. Midway through the outcome study, one company 
relocated its operation to another plant and another was unable to modify its 
education plan to comply with federal guidelines. As a result, Evaluation 
Research conducted termination evaluations in these two companies. The 
"termination study" became the third section of the MWEI Cycle 4 Evaluation. 

Background of the MWEI and the Cycle 4 Evaluation 

The Massachusetts Workplace Education initiative was begun in 1985 as an 
interagency effort to bring adult basic education and English as a Second 
Language (ESL) instruction directly into workplaces throughout the state. State 
policy makers had at that time become concerned about the large numbers of 
workers who did not have the basic skills to apply successfully for new jobs after 
plant closings and other lay-offs. This concern led the former Office of Training 
and Employment Policy in the Executive Office of Economic Affairs, the Bureau 
of Adult Education in the Department of Education, the Executive Office of Labor 
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and, later, the Commonwealth Literacy Campaign to form a partnership to 
improve workers' skills before they lost their jobs. It was felt that a more-skilled 
workforce would strengthen business, enhance competitiveness in a harsh 
economy, and save jobs. 

As this workforce initiative got underway, it was becoming dear that 
Massachusetts was not atone in its concerns about the readiness of its 
workforce. Increasingly, a national alarm was being sounded in the late 1980s 
about the need for American employers and union leaders to take more 
responsibility for re-skiHing the workforce. Prominent figures like Labor 
Secretary Elizabeth Dole and Xerox CEO David Keams alerted the nation to the 
education "crisis' in the workforce. "Workforce 2000; Work and Workers for the 
21st Century," the widely-cited 1987 report by the Hudson Institute, stated 
unequivocally that as the American workforce grows older, includes more 
women, and becomes racially and ethnically more diverse toward the year 
2000, no institution in the nation can automatically depend on having an 
unending stream of new workers with the right skills to match available jobs. 
Further analysis of the problem and possible solutions were offered in 
"America's Choice: High Skills or Low Wages," the challenging report of the 
National Center on Education and the Economy. The authors described the 
mismatch between an aging American workforce trained in large-lot, assembly- 
line manufacturing processes and the demands of a new. global, customized 
manufacturing economy. They suggested that America increase its productivity 
by reorganizing how we work, training workers to perform more skilled jobs, and 
emphasizing quality. 

Organized labor was also adding its voice to the discussion with a similar set of 
arguments. Anthony Sarmiento, Assistant Director of Education for the AFL- 
CIO, wrote: "If managers focus entirely on their firm's next quarter performance, 
long-term investments like worker training and research and development are 
ignored. How do we avoid the economic 'cliff ahead, where the gap between 
the economic 'have's' and 'have nots* will widen still further and social tensions 
will deepen? Our first task is to better understand the nature of the problems we 
face. We need to restructure our outdated workplace, not just expand and 
improve our inadequate education and training systems. Both will require 
money and patience, as well as a renewed commitment to all workers and their 



families." Numerous other reports, studies, and articles were also being written 
by analysts from varied fields which legitimized the need for improving workers' 
skills as a major component of a larger reorganization of the American 
workplace. 

Local business leaders took up this cause, as well, in early 1991 twenty-three 
Massachusetts CEOs sent a letter to Governor William Weld urging state 
support for education in the workplace. They said: "We are, all of us, 
dependent upon the quality of our workforce to make quality products and to 
deliver quality services on time. We cannot compete in today's international 
markets unless we can offer the high quality goods and services our customers 
demand, when they demand them. To do this we must have a workforce skilled 
for the '90s; and we must have it now, not ten or twenty years from now." 

However, despite these calls to action, information remains relatively scarce 
about how education programs function optimally in the workplace, how 
curricula are best developed and implemented, and what is actually being 
achieved in these programs. 

Former Evaluations 

Cycle 1 

In the mid-1980s, the MWEI state partnership found very little information to 
guide its development of service delivery and policy in the new field of 
workplace education. As a result, the partnership commissioned a series of 
evaluations which documented the varied workplace education service delivery 
models being developed around the state. With this information, the state 
partnership began to systematically educate itself about this new enterprise. 

In Cycle 1 , the evaluation consisted of ethnographically-oriented case studies of 
six workplace education programs. In order to understand how workplace 
education programs might function in different settings with different worker 
populations, six sites were selected which represented a cross-section of 
industries and populations: a distribution center for a national discount retailer 
of clothing and household goods, a hospital, a small metals manufacturing 



company, two laundries, a national communications corporation, and a 
consortium of clothing manufacturers. 

The case studies documented how the programs were conceived, developed, 
and carried out on a day-to-day basis. They described the worker population, 
the learner population, recruitment methods, curriculum, scheduling of classes, 
barriers to program participation, and particular program strengths and 
weaknesses. Out of these six case studies emerged the basic guidelines for 
effective program development that the MWEI has built on over the years. 
These included: 

• Conceive of your workplace education program as a partnership that 
represents the varies interests of the partners - upper-level management, 
supervisors, union representatives, workers who are enrolled in the program, 
education staff - even workers who are not enrolled in the program. 

* Do not assume that the support of upper-level management atone will insure 
the success of a workplace education program. Involving mid-management 
and supervisors in program planning is also essential. These are the people 
who have daily contact with the workers and who therefore support - or do not 
support - the program in practice. 

♦ Designate a program coordinator whose job it is to facilitate the partnership 
and oversee program development and implementation. 

Cycles 2 and 3 

The Cycle 2 evaluation continued to build a data base about what was going on 
in MWEI programs by preparing detailed case studies on an additional 
seventeen programs. It also continued to identify the essential "building blocks" 
of a successful workplace education program. But by Cycle 3 policy makers 
were looking for information which focused more specifically on program 
outcomes. 

From the beginning, the state had seen its investment in workplace education 
as a way of "seeding" programs whose costs would eventually be picked up by 
the host employer and/or union. By Cycle 3 it was becoming increasingly clear 
that employers wanted to know whether the programs were delivering results or 
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not before they invested in them. Similarly, unions were exploring the 
conditions in which they would or could support basic skills programs with 
money as well as good will. 

As these potential "investors" in workplace education looked more closely at 
where to put their worker education resources, questions persisted about how 
to define and measure success. For example, some employers and unions 
began to see education as one part of an overall strategy to improve production, 
services, and the organization of work, and believed it was misleading to 
assess their education program apart from other changes being undertaken. 
Other institutions were more tied either to traditional academic measures of 
student learning (measures whose applicability in the workplace were 
questionable), or to a "hard" measure of cost saving that would guarantee return 
on investment. 

In response to these questions about the desired outcomes of workplace 
education and how to measure those outcomes, the Cycle 3 evaluation 
negotiated with seven workplace programs representing different industries and 
worker groups to participate in a pilot outcome study to be conducted by an 
outside evaluator. Four of these programs were in small businesses, one in a 
hospital/nursing home complex, one in a high-tech manufacturer of 
communications technology, and one in a distribution center for a national 
discount retailer of clothing and household goods. All seven of the programs 
provided ESL classes and only one provided primarily adult basic education. 

This Cycle 3 pilot outcome study was the MWEI's first attempt to (1) identify and 
measure site-specific outcomes, (2) analyze what outcomes are meaningful in 
different business and industry contexts, and (3) build a case for why 
companies and unions should continue to support education beyond the 
"seeding" stage. 

The pilot outcome study found that: 

• Outcomes are relative to the goals which are established for a program. A 
workplace education program is a partnership among different interest 
groups which may hold different goals for the program, including, at least: 
the participating workers, union representatives, supervisors, and managers. 



Complete and fair evaluations of outcomes must therefoie provide all 
players opportunities to, first, define the outcomes to be evaluated and then 
assess what outcomes in fact are being achieved. 

• Outcomes can be desired or actual. Desired outcomes are what the program 
wants to bring about, especially in the long term; and actual outcomes are 
what the program is in fact achieving in the present. The actual outcomes 
can be anticipated, or unanticipated . Anticipated outcomes are those that 
are consonant with the desired long term outcomes. Unanticipated 
outcomes are surprise, unpredicted outcomes. In the case of the workplace 
education programs studied in Cycle 3, unanticipated outcomes were 
universally positive and shed light on how broadly defined the effects of a 
workplace education program might be. For example, entry-level workers in 
a nursing home reduced their absent days to almost zero, saving the 
administration thousands of dollars in fees to agencies who provide day- 
replacement workers. Similarly, a small manufacturing company found that 
the number of accident claims made to the personnel office was reduced to 
almost zero. 

• Desired and actual outcomes vary from program to program, even within the 
same industry, and they also vary across industries. Outcomes are relative 
to the type of basic skills, ESL or other educational service offered, 
educational staff and materials used, the demographics of the worker 
population, whether a union is involved, the type of product being produced, 
management styles, and the economic conditions in which the company 
competes. Workplace education program outcomes are, therefore, not fixed 
entities which can be approached in a one-dimensional way. Rather, 
outcomes change from program to program, and even within programs 
themselves, depending on a variety of conditions. 

The Cycle 3 evaluation thus helped the Partners clarify some of their concerns 
about how to measure program outcomes. Wanting to build on these findings, 
the MWEI was, in Cycle 4, motivated even more strongly to help programs 
continue to assess their outcomes in terms that are more s pecifically relevant to 
employers', unions', and learners' goals . As a result, the evaluators developed 
a process for defining and measuring meaningful, site-specific outcomes which 
was piloted in six workplace education programs. This process would be 
designed so that any workplace program could adapt it to its own evaluation 
needs and resources and develop its own ongoing, customized evaluation 
system. Evaluation would thereby become seen within MWEI not as something 
done "to" programs by outside evaluators but as an intrinsic part of every 
program's day-to-day operations. In Cycle 4, the Partners also recognized the 
need to have more-detailed information on the range of curriculum models 
being developed in its programs. MWEI thus took the opportunity to review the 



organization and delivery of educational materials and curricula and to assess 
the overall effectiveness of these materials and curricula. The curriculum study 
forms the second section of this report. 

Participating Programs 

The following programs participated in the Cycle 4 curriculum and outcome 
studies. The curriculum study was conducted in four of the same programs that 
hosted tha outcome study. An asterisk indicates participation in both studies. 
No asterisk indicates participation in the outcome study only. The programs at 
Friction Materials in Lawrence and Spir-it, Inc. in Wakefield are described in the 
termination study, section three of this report. 

Laura Sperazi conducted the termination study. She also conducted the 
outcome study at the Hampden Paper Company and Aerovox, Inc. Paul Jurmo 
conducted the outcome study at the United Electric Controls Company, 
Kennedy Die Castings, Inc, South Cove Manor Nursing Home, and Boston City 
Hospital. David Rosen conducted the curriculum study at the South Cove 
Manor Nursing Home, Boston City Hospital, Kennedy Die Castings, Inc., and 
the Hampden Paper Company. 

Hampden Paper Company, Holyoke* 

Hampden Paper is a small (180 workers), family-owned, paper 
manufacturing company situated along the Connecticut River. As the work 
at Hampden Paper becomes more skilled and more customized, greater 
demands are made on workers to communicate, collaborate, problem-solve, 
innovate, perform quality control, and perform more than just one job. 
Education.is seen as a primary strategy for sustaining this massive change 
at Hampden Paper. 

At the present time, ail levels of ESL, ABE and GED instruction are offered 
on an individualized, by-appointment basis in the company learning center 
(classroom). In addition to the instruction currently offered, a "certification" 
program will soon be developed. This will mean that workers can become 
certified to perform as many operations in the production process as they 
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choose to team through enrolling in ~ and completing -- job certification 
programs. 

Aerovox, Inc., New Bedford 

Aerovox, Inc. manufactures capacitors - conduits for electrical charges. 
Aerovox Is a medium-sized company (about 700 employees) in a mature 
industry faced with the challenge to automate, down-size, develop a quality 
control program, and to improve worker skills. These changes must be 
made if the company hopes to compete with companies which are 
producing less expensive, equally sophisticated products outside the U.S. 
As the company embraces more sophisticated production technology, the 
demand for workers to communicate and read well in English and to perform 
math functions required for statistical process control increases substantially. 

The three components of the Aerovox workplace education program - ESL, 
ABE/GED, and math for SPC - all directly address the need for improving 
workers* basic skills to prepare them for new forms of work. When 
management looks to where it wants the company to be six years from now, 
education becomes a larger and larger part of the overall strategy to get it 
there. 

Kennedy Die Castings, Inc., Worcester* 

Kennedy Die Castings, Inc. is a forty-one year old manufacturer of metal 
castings. It employs about 100 people. Like many small manufacturing 
companies throughout the state, Kennedy Die Castings faces new 
challenges as it: converts its production methods to include sophisticated, 
computerized machinery; introduces the team approach to problem-solving 
known as "quality circles;" and institutes statistical process control to keep 
costs down and quality up. 

About 65 % of the employees at Kennedy Die Castings are limited English 
proficient. Most of these employees are Southeast Asian, Polish, and 
Hispanic. Changes in the production process demand that these workers 
adapt and upgrade their skills to accommodate increasingly sophisticated 
machinery and team methods of problem-solving. The company offers one 
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ESL class. Providing on-site instruction is one step toward developing a 
confident, skilled and adaptable workforce. 

United Electric Controls Company, Watertown 
Since 1987 this successful family-owned manufacturer of industrial sensors 
and temperature control devices has overhauled Its management and 
production technologies according to the principles of "continuous 
improvement." Motivated by severe profit losses, it abandoned traditional 
"large tot" manufacturing methods in favor of techniques that stress 
employee participation in the production process. In order to facilitate these 
company-wide changes, management offered ESL instruction to its limited- 
English-proficient workers, approximately 45 of its 350 employees. 

The ESL program continues to be an integral part of the overall strategy to 
continually refine the production process. Without it, old and new 
employees who have limited English language skills would not be able to 
participate fully in these changes. 

South Cove Manor Nursing Home, Boston* 

The "product" which staff produce in this one-of-a-kind nursing home for 
elderly Chinese people in the Commonwealth is quality resident care. 
Differences in culture and language between primarily-Chinese nurses' 
aides and other entry-level workers and native-English-speaking 
supervisory staff can compromise quality of resident care in serious ways. 
For example, the seemingly small error of placing a salted or sugared food 
item on a food tray for a resident with restricted dietary needs may cause 
health problems. 

The South Cove Manor ESL program facilitates communication among 
English-speaking R.N.'s and M.D.'s, Chinese-speaking nurses aides and 
other entry-level workers, and between nurse's aides and residents, thereby 
improving the quality of resident care. 

Boston City Hospital, Boston 4 

The main facility of this large urban medical center which serves Boston's 
poor recently underwent major renovations in order to provide better care to 



more people. Like other hospitals and health care agencies, Boston City 
Hospital (BCH) faces a critical shortage of trained workers at the entry, aide, 
technician, and R.N. levels. Basic skills instruction and a unique study skills 
course is one part of the hospital's strategy to retain loyal workers, promote 
them through the ranks, and become a quality care facility. 

The BCH program Is unique in two ways. First, it recruits workers from three 
other affiliated health care facilities through union membership. (There are 
three unions which support the BCH workplace education program: 
AFSCME 1482; SEIU 1475, and SEIU 285.) And, second, It offers two very 
different levels of instruction: the ABE/PRE-ASE/ASE level and a study skills 
class which prepares people for study beyond the secondary level. 

These programs are described in greater detail in the case studies contained in 
Appendix A, under separate cover. 
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The Outcome Study 



The Evaluation Team Process: How K Answers Difficult Questions in 
Workplace Education Program Evaluation 

In a widely-cited article titled "Evaluating National Workplace Literacy 
Programs," Thomas Sticht defines the purpose of a workplace education 
program evaluation. His frame of reference is programs funded through the 
National Workplace Literacy Program (NWLP). He says that program 
evaluation should be seen in the context of the goals of the NWLP: "The 
ultimate goal of the National Workplace Literacy Program is to improve the 
productivity of the workforce through the improvement of workers' literacy 
abilities. For this reason, after providing convincing evidence that 
improvements have taken place in literacy abilities, the workplace literacy 
provider needs to present convincing evidence that the improvements in literacy 
have led to improvements in job productivity." (p.10) In the same article he also 
says that: "...one of the goals of evaluation is to permit the improvement of 
programs, not to simply decide if they work or not." (p. 10) 

While educators, union representatives, workers enrolled in programs, and 
even some company managers might choose to define the ultimate goal of their 
programs in somewhat broader terms than improving the productivity of the 
workforce, the fact remains that demonstrating a "convincing" link between 
worker participation in a workplace basic skills program and increased job- 
related productivity is a nation?' priority. The task of developing meaningful 
measures of that "convincing link" is a difficult one which requires real, ongoing 
collaboration among the partners represented in a workplace education 
program: employers, unions, educators, and workers themselves. One vehicle 
for building this collaboration is an "evaluation team" in which these interest 
groups together identify the outcomes which they hope their education 
programs will achieve, measure what outcomes are, in fact, being achieved, 
and demonstrate the link between the education program and increased 
productivity and other desired outcomes. 

It took the Massachusetts Workplace Education Initiative four cycles of 
evaluating its programs before it began to explore this process of team 
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development and mutual definition and measurement o* outcomes. Despite the 
Imperfections of the process - it is new and needs considerable refinement - it 
took us one big step closer this year to actually forging and demonstrating a link 
between basic skills instruction at the workplace and increased productivity. 
Furthermore, the evaluation team has became a vehicle through which the 
information gathered in the evaluation can be used to improve program 
services. Thus, evaluation becomes more than just figuring out what works or 
not.- it becomes a way of using what you learn to improve what you do. 

The MWEI Outcome Evaluation: Cycle 4 Summary Results 

The following is a summary of the findings of the MWEI Cycle 4 Outcome Study. 
Subsequent sections of this report describe in detail the ten-step process we 
employed and elaborate further on the study's findings and future directions. 

• The main finding of this study is that it is both possible and desirable to 
establish a process for identifying and measuring workplace education 
program outcomes which actively involves representatives of the entire 
partnership. This requires a major commitment of time and hard work from the 
company, union, education staff, and students which should be built into the 
work-plan for the project. Outside consultants - acting as facilitators of a 
team-building and planning process -- are an essential element in the 
establishment of the evaluation team, especially in the beginning stages of the 
evaluation. 

• In this model, consultants are really training the teams in many aspects of 
survey and evaluation research - goal-setting, questionnaire construction, 
data analysis, and so forth. If team members have some familiarity with the 
mechanics of evaluation, the work is made easier. If team members have 
limited - or no - familiarity with evaluation, problems will then no doubt arise 
about the amount of time spent in this work, and who (the team or the 
consultant) should be doing it. 

• Designing data-gathering instruments is a difficult task that requires expert 
help for two reasons. First, people want instruments that take very little time 
but are also accurate. Second, the way that we typically think about data- 
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gathering instruments in this setting - questionnaires, surveys and checklists - 
- might be time-saving but not the best way to measure what needs to be 
measured. Simulations, indirect measures of achievement, use of existing 
data sources tike productivity or personnel records - these and other 
measurements should be explored. Establishing a computerized data base 
with a software package like Statistical Program for the Social Sciences 
(SPSS) early in the evaluation can help to organize how team members think 
about evaluation and helps team members to streamline questionnaires and 
surveys. 

• Student self-report data are central to understanding the change that takes 
place in a workplace education program, and no evaluation of a workplace 
education program would be complete without them. There are even some 
workplace education programs (not in the group of programs under study in 
this report) which consider student self-reports the most reliable source of 
data about program impacts. 

However, this year's new look at program outcomes raises old methodological 
questions about (1) the strengths and limitations of student self-report data, 
and (2) about the disparity between students' and supervisors' perceptions of 
change. Much of the data collected by the participating programs are student 
self-report data supplemented by supervisor reports, and, in one case, by 
"customer reports from the residents in the South Cove Manor Nursing Home. 

In some cases, when student self-report data were compared with 
supervisors' assessments of those same students, there turned out to be a 
significant disparity between supervisors' and students' perceptions. This 
disparity presented the consultants with a number of questions: "What does 
this disparity mean?" and "Whose assessments more accurately reflect reality 
-- the supervisors' or the students'?" 

Upon further reflection, the consultants identified the following factors as 
possible reasons for the disparity between supervisors' and students' 
assessments: 



13 



Different expectations or standards : Students and supervisors might be 
operating with different levels of expectations about what students should 
be achieving. For example, a limited-English-proficient nurse's aide 
might consider it a great achievement to simply be able to say a few 
words of greeting in English to patients, white a supervisor might expect 
that same aide to be much more fluent. Furthermore, the supervisor 
might be looking at the employee's overall competency rather than 
facility with the components of a learning task. For example, the student 
may have learned many of the steps involved in using fractions and see 
tremendous improvement in his math, but not yet be able to apply that 
learning to his job. 

Lack of understanding of an evaluation's terminology or "mechanics": 
Some students and/or supervisors might not fully understand the terms 
being used in, say, a questionnaire. Or perhaps they arent familiar with 
the rating scale used. If frustrated, embarrassed, or confused by these 
technicalities," an informant might provide inaccurate responses to the 
questions asked rather than take the time -- and risk embarrassment to 
ask for help. 

Lack of trust a bout how responses will be used: Students might fear that, 
if they give themselves a low rating on a self-report, they - or their 
teacher - might somehow be punished. Similarly, a supervisor might be 
concerned that giving a student a "low" rating would jeopardize that 
student's chances for promotion; that supervisor might therefore tend to 
inflate students' scores. 

Comparing aggregate ratings by su pervisors to individual self- 
assessments bv students : When there are many students enrolled in a 
program, with significant numbers in each department in the host 
institution, it is very time-consuming for supervisors to assess change in 
students on a case-by-case basis. Consequently, in some programs, 
supervisors were asked to respond to questions about change in 
students' behavior in the aggregate. That is, they considered the 
students as a group rather than as individuals. While these aggregate 
data give overall impressions of what changes have occurred in 
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students 1 performance - data which can be useful - they 'flatten" the 
assessment of change by making it impossible to distinguish between 
those students who have learned/changed considerably and those who 
have teamed/change less. When data from students' self-assessments 
are compared to the composite picture presented in the supervisors' 
assessments, there will naturally be a disparity between the students 
who rate themselves tow or high and the "average" scores reflected in 
the aggregate data. 



It appears that student self-reports are most useful under the following 
conditions: (1) students are actively involved from the start in setting program 
goals and designing meaningful and "user-friendly" instruments for gathering 
data about actual progress toward those goals; (2) students have an 
opportunity to practice using those instruments, so that they understand the 
"mechanics" of the instruments as well as the need for honest, thoughtful 
answers; (3) there is a high degree of trust between students and others 
involved in the program, so that ail involved support the notion of students' 
providing honest, thoughtful information, and (4) students' self-reports can be 
directly compared to data from supervisors' ratings and/or other sources. 



• All the teams were interested in identifying and measuring outcomes which 
would demonstrate what the link was between their workplace education 
program and specific improvements in employee job performance. These 
outcomes, in general, consisted of improved abilities to apply reading, writing, 
oral communication, and math in specific job functions. However, all the 
teams were also interested in broader outcomes which represent a new order 
of skills demanded in the changing Massachusetts (and American) workplace: 
for example, problem-solving, taking initiative, and working effectively with a 
team. Many of the evaluation teams were also interested in outcomes that 
focus on students' personal development (for example, going on to college, 
being able to communicate with neighbors), on their understanding of their 
rights as workers and citizens, and on their participation in their unions. All of 
these skills were seen as complementary and integral to the "whole" learning 
which makes adapting to new challenges possible. Thus, the pool of 
outcomes identified by the teams reflected an understanding of how an 
employee's productivity can be strengthened by helping the worker to develop 
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a mix of skills at the same time: immediately-job-specific skills, broader skills 
transferable across a number of jobs, and "life skills" which not only enhance 
the employee's personal development but are transferable back to the job. 

• Despite the methodological limitations of this study, evaluation teams did 
report change. For example, at Hampden Paper, understanding production 
tickets is a critical part of the production process. These "tickets" give 
instructions on how to make a particular paper product. Sixty percent of the 
supervisors who responded to the supervisor questionnaire noted 
improvement in workers' accuracy in reading these tickets. Eighty-soven 
percent of the students who responded to the student questionnaire noted 
improvement in reading accurately. 

At Aerovox, an identified - if unanticipated - improvement in workers' taking 
safety precautions may save the company a significant amount of money by 
reducing the amount of worker's compensation insurance which the company 
needs to purchase. (Over 81% of the workers noted improvement in their 
safety-related behavior, as did 66.7% of the supervisors.) While the 
workplace education program may not claim alt the credit for this improvement 
- the company nurse had also begun more safety training (especially with 
drivers) during the year, and reassigned injured workers to "lighter" jobs - the 
members of the evaluation team, including upper-level managers, believed 
that the education program was, at least, partially responsible. 

The exhaustive tabulation of the data gathered by the South Cove Manor 
program coordinator provided similarly telling insights into the outcomes a 
workplace education program might bring about. The education program is 
credited with significantly increasing positive interaction between residents 
and nursing assistants. For example, residents feel they interact positively 
with 91.6% of students; residents report that 83.3% of students adequately 
explain procedures; residents report that they now interact more positively 
with 66.6% of students; and students feel that they now satisfactorily engage 
in 57.81% of the positive interactions considered critical to quality care. 

At Kennedy Die Castings, the supervisor who twice evaluated the students 
noted that students, on average, improved slightly during the period of June to 
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August in their abilities to handle all of the given job tasks, and that they 
showed the strongest abilities in (a) identifying quality defects, (b) functioning 
as a Kennedy employee, and (c) comprehending and following directions. 



Chart A: Summary of Types of Outcomes Identified In MWEI Cycle 4 Outcome Study 
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X 


Aerovox 
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X 
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X 


UE 
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X 



17 



Chart B 



SUMMARY OF DATA-GATHEF 


KING ACTIVITIES UN 
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Hampden 
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Aerovox 
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1 0 Ls ( twice) 
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4 sups, rated 1 4 Ls 
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8 sups, rated 1 2 Ls 
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8 clients rated 12 Ls 
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6 sups, rated 6 Ls 
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TOTALS 


34 sups, rated 107 Ls 


1 sup. rated 1 0 Ls ( twice) 


134 Ls 


19 Ls 


8 clients rated 12 Ls 


21 Ls 
















Abbreviations: 












Q= questionnaire 












Sup. = supervisor 












Ls - learners 













The Outcomd Study as a Ten-Step Process 

The outcome study is reported here as a ten-step process that both reports on 
the work of the participating programs and offers a schematic which - with 
adaptation can be used in other programs. Note that the data-gathering 
instalments used and the findings at each site are described in greater detail in 
the case studies contained in Appendix A, under separate cover. 

I. Goals to be achieved: 

The outcome study was designed as a combined outcome study and 
organizational development project. The project attempted to achieve the 
following three major goals: 

1. Build the capacities of individual programs to conduct outcome 
evaluations. 

2. More clearly define desired outcomes for each site and produce data 
which demonstrate what in fact is being achieved. 

3. Develop a "tool kit" (an evaluation planning process, data gathering 
instalments, and resource persons) which other programs can use to 
develop their own planning and evaluation systems. 

This was thus not a traditional "outcome study" conducted by outsiders but 
rather an effort combining organizational development and outcome evaluation 
activities. The consultants discussed these goals with the teams throughout the 
course of the project, to make it clear to the teams what in fact the project was 
attempting to achieve. 



II. The ten-step process used in the six sites: 

The process used in the outcome study consisted of a series of: 

1. meetings between the consultants and the six evaluation teams; 

2. individual discussions between the consultants and team members; 

9 3. 




3. independent work by individual team members -- especially the 
program coordinators - and the consultants to develop and use 
evaluation instruments; 

4. a special workshop on how to conduct an outcome evaluation; and 

5. a special workshop on uses of computerized data-processing systems. 

in the meetings between the teams and the consultants, the teams were 
typically presented with a set of open-ended questions, or set of goals to be 
achieved, which aimed at getting them to think through - and define for 
themselves - the central issues which need to be addressed when planning 
and implementing an outcome study, in this approach, the consultants 
intentionally avoided prescribing a standardized formula for all teams to adopt. 
This approach aimed instead at helping the teams to develop evaluation 
systems customized to their unique needs and resources. 

The consultants and teams went through the following ten-step process over the 
period of late 1990 through the fall of 1991 : 

Step 1: Evaluators familiarize themselves with the programs: December 
1990 - February 1991 

The consultants familiarized themselves with the history of the programs 
through review of documents from previous evaluations, interviews with 
program coordinators, and through site visits to the workplaces themselves. (In 
some cases, tours of the organization/program occurred during the first site 
visits described in Step 4 below.) 

Step 2: Program coordinators convene their evaluation teams and inform 
them about the evaluation which is planned for the coming year: December 
1990 - January 1991 

Each program funded under the MWEI is headed by a coordinator, typically a 
representative of the educational institution providing the actual education 
services. The coordinators to be involved in the Cycle 4 evaluation had agreed, 
in conversations with the consultants, that team members should represent the 
various interest groups involved in the workplace education program 
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partnership -- workers enrolled in the program, education staff, union 
representatives, supervisors, and managers. To prepare for the work which the 
team would undertake during the course of the year, the program coordinators 
discussed the evaluation team concept with potential team members and then 
invited them to a preliminary meeting. In this preliminary meeting, the 
coordinators presented the goals of the evaluation, and prepared the team for 
the meeting of all the teams from the participating programs to be held in Boston 
on January 10th. 

Step 3: Teams from six sites meet as a group: January 10, 1991 

Representatives from all six sites met in Boston on January 10th. There they got 
to better know their fellow team members and were introduced to people 
representing similar interest groups in other workplace education programs 
from around the state. This was the first time that members of the various 
workplace education interest groups had the opportunity to meet with each 
other across programs. The consultants gave participants an overview of the 
above-described purposes of the outcome study along with a proposed 
schedule of activities for the subsequent nine months. 

The teams were then challenged to work in various kinds of group activities to 
begin clarifying for their respective sites answers to the following fundamental 
question: "Who wants to know what information for what purpose?" (This 
question is central to an evaluation process developed by Professor David 
Kinsey of the Center for International Education at the University of 
Massachusetts at Amherst. See Kinsey in References) 

More specifically, this question was broken down into sub-questions: 

1 . Who are the interest groups represented in your program? 

2. What information do those various interest groups want to develop in 
this evaluation? 

3. Why do they want to know that information? 
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These discussions produced the following kinds of statements: 

1. "Who?" : The key interest groups represented in the evaluation teams 
were upper-level management, supervisory personnel, unions, adult 
education staff, and employees participating in the program. Other 
parties not directly involved in the teams - but who nonetheless might 
have an interest in knowing what the program was accomplishing -- 
were program funders, the families of participating employees, the 
communities of those employees, and workers not participating as 
students in the program. 

2. "What information?'' Initially, team members tended to say they wanted 
to know how the "self-esteem," "communication skills," and 
"productivity" of employees were impacted by the education programs. 
When encouraged to be more specific, they were able to give some 
examples of more-specific results - particularly job-related abilities - 
which they wanted to examine. 

3. "Whyy The teams said that, overall, they wanted to know whether the 
program was "working." When pressed to be more specific about 
"why" they wanted to know this, they cited two key reasons: (a) to 
demonstrate to funders and management the value of the program, 
and (b) to clarify what needs to be done to improve the program. 

Step 4: Each team further defines "Who Wants to Know What Information 
for What Purpose and How Will that Information Be Gathered?": February - 
March 1991. 

The consultants at this point took essentially the same approach to getting the 
teams to further define what information they wanted to get in the study and how 
they would get it. However, because there was some variation in the two 
consultants' interactions with the evaluation teams, the two consultants' 
activities will here be described separately. 

The two consultants took slightly different approaches to their work with the 
teams for the following iasons: 



1. Two programs had not participated in any prior MWEI evaluations. 
Therefore, the amount of information already gathered about these 
programs was limited. This made it necessary for the consultants to take 
extra time to develop basic understanding of the issues and players at 
these sites before the work could really get underway. 

2. Both consultants saw a value in making their interactions with the 
evaluation teams as "natural" as possible. This meant that each 
consultant was to use his or her best instincts and prior experience and 
try to respond to the unique interests and strengths of each team. It was 
felt that such a "natural" process would produce a more valid picture of 
how evaluation teams can work than would a team effort shaped by a 
predetermined, artificial set of "rules." 

3. One consultant, as a resident of Massachusetts, had more-ready 
access to the program teams than did the second consultant, based in 
New Jersey. This gave the Massachusetts-based consultant a freer hand 
in scheduling site visits and otherwise communicating with the teams, 
thereby allowing her to be more flexible and responsive vis-a-vis the 
needs of each site. 

At Hampden Paper and Aerovox, the consultant structured the evaluation 
activities more as a series of tasks the team had to accomplish. At South Cove 
Manor, United Electric Control, Kennedy, Boston City Hospital, the consultant 
structured the evaluation activities more as a series of questions the team had 
to answer. The tasks to be accomplished or the questions to be answered were 
comparable for each of the visits which the consultants made to the sites. 
Having completed the evaluation team process with these slight variations, the 
consultants agree that structuring the on-site meetings as as series of questions 
helps the team to focus on the work to be done, especially when team members 
are involved in evaluation work for the first time. 

First Meeting with Hampden Paper and Aerovox 

Each of these programs posed special challenges to the outcome evaluation 
which led the consultant to create special activities for the initial site visit at 



these two sites. Neither program had participated in a MWEI evaluation before. 
Consequently, there was no base of information on which to build the outcome 
study, and no prior experience for team members of working with an external 
evaluator or with an evaluation process that involved all the key players in the 
partnership. Furthermore, the Aerovox program was iess than a year old, still 
developing its "sea legs" and had been unable to send a full team to the 
January 10 workshop. (Only the program coordinator and the Vice President for 
Human Resources attended.) 

These conditions required the evaluator to use the first site visits to both 
programs as a way of developing an understanding of who the workers are, 
what/how the company produces, and how the workplace education program 
fits with company needs and goals. The site visit also served to build interest 
and trust in the new experience of carrying out their own outcome evaluations. 
The goal of the first team meeting at each of these programs was for these 
reasons threefold: 



(1) to gather basic information about the programs; 

(2) to increase the programs* interest in carrying out their own outcome 
evaluations; and 

(3) to generate a working list of outcomes for each program and possible 
methods for measuring them. 

At Hampden Paper, the consultant spent day #1 interviewing the company 
president and program coordinator, touring the facility, and conducting the first 
on-site meeting of the evaluation team where she reviewed the "Who," "What 
Information," and "What Purpose" schematic which was used in the January 
workshop as well as the answers to those questions which the Hampden team 
had generated. The consultant also attended an advisory board meeting where 
the evaluation plan was reviewed and approved. 
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At Aerovox, the consultant spent day #1 interviewing company managers and 
the program coordinator, touring the facility, and replicating on a small scale for 
the whole team the workshop described under Step 3. The Aerovox team was, 
in effect, introduced to the "Who," "What Information," and "What Purpose" 
schematic during this first site visit. 



A fourth question of "How will information will be gathered?" was added at this 
point in the presentation to both teams, as a way of getting them to think through 
what kinds of data-gathering instruments they might develop. By asking both 
teams to develop answers to these questions, the consultant helped team 
members to: 

(1) clarify the outcomes which are meaningful to their programs; 

(2) begin to think about the instruments and processes which would best 
answer their questions; 

(3) begin to think about who would collect, analyze, and report the data. 

In both companies the teams brainstormed freely to identify what significant 
outcomes in their programs might look like. The consultant helped the teams to 
refine and narrow their definitions and recorded them. 

Hampden Paper 

At Hampden Paper the evaluation team identified many outcomes which were 
either job-specific, of particular interest to the union, related to program 
development, or related to fostering an environment of greater support for 
education as a precursor to the certification program. These are the outcome 
questions which the team members chose to focus on: 

• Do workers readily enroll in and attend classes, and if not, why not? 
(This question will have more relevance as the certification program 
gets underway. Enrollment and attendance in classes will be seen as 
valuable outcomes in themselves. Reasons for lack of enrollment or 
attendance will be critical variables for program staff.) 

• Do students move up the job ladder in the company as a result of 
participating in the education program? (This question will have more 
relevance as the certification program gets underway because one 
goal of the certification program is to facilitate promotion.) 

• Do students read production tickets accurately? 

• Do students read measuring devices accurately? 

• Do students write accurate work reports? 

• Do students ask more questions on the job? 



• Does accurate reading and reporting impact scrap reduction? 

• Do students: 

- know the benefits that are available through the union? 

- understand the grievance procedure ? 

- identity problems on shop floor as union concerns? 

- understand the union contract? 

- have an interest in becoming a steward? 

- understand the union contract? 

- understand how the union supports workplace education? 

Aerpypx 

in this introduction to program evaluation in general and outcome evaluation in 
particular during the first site visit, the team at Aerovox found it somewhat 
difficult to distinguish between program/process variables (like involving 
supervisors in the program ) and outcomes. The consultant realized that it was 
difficult for any team - particularly a new one like that at Aerovox - to in one 
meeting both learn about the fundamentals of outcome evaluation and actually 
begin planning the details of such an evaluation. The consultant therefore used 
the initial Aerovox meeting as an opportunity for team members to begin 
brainstorming responses to the "Who, What Information, Why, and How" 
questions. In subsequent weeks the team, under the leadership of the program 
coordinator, had additional time to further refine their answers. Within a few 
weeks, then, the Aerovox team had become clearer about the distinction 
between "process" issues and "outcome" questions. They came up with the 
following job-specific outcome questions: 

Can workers who are students: 

• describe their jobs? 

• inspect parts of their equipment? 

• practice maintenance of their equipment? 

• practice cleanliness of equipment/surroundings? 

• take safety precautions? 

• understand the purpose of having an MPI (manufacturing process 
instruction)? 

• follow an MPI from their own section? 
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• follow MPIs from other sections? 

• read graphs? 

• construct a graph? 

• understand SPC procedures? 

• plot an SPC chart? 

• understand their "right to know"? 

Both teams agreed that the data should clarify how the program might be 
improved and also be used to help the program argue effectively for funds from 
potential funders. Neither company needed the data to demonstrate internally 
whether the investment in education was worthwhile or not in order to continue 
the program. Both began from the position that education is worthwhile, had 
made substantial cash investments in their programs, and were planning to 
continue that investment into the coming year. However, both teams were 
interested in understanding the more-than-obvious ways in which their 
respective investments in education were paying off. 

The teams agreed that the information should be gathered through checklists, 
surveys, and interviews. These instruments would be used with supervisors, 
union leaders, and, most importantly, students themselves. The teams were 
concerned that the data collection not be too time-consuming. In addition, data 
would be gathered from already existing sources at each site. At the conclusion 
of the first team meetings at Hampden Paper and Aerovox, the consultant 
agreed to draft samples of protocols which would reflect the outcomes 
described above and send them back to the teams for revision. 

First Meeting with Kennedy Die Castings. South Cove Manor. Boston Citv 
Hospital, and United Electric Controls 

Based on the program information developed in Steps 1 and 3, the consultant 
prepared a "grid" listing "Who," What Information,** "What Purpose," and "How" 
as it might relate to an outcome study for each of the four teams he was to work 
with. The grid showed possible interest groups ("Who") to be involved, what 
type of information ("What Information") they might want to know, "What 
Purposes" would be achieved by getting that information, and what kinds of 
data-gathering activities ("How") might be used to get the data. Each team 



(especially the coordinator of each team) was asked to review that team's grid, 
to get a clearer idea of what the outcome study might encompass and to be 
prepared to add or delete details from the grid to make it more relevant to the 
team's situation. 

In site visits during the period of February 21st to March 6th, the consultant met 
for 2-3 hours with each of the four teams. Each team was asked to now "fill in 
the blanks" in its grid, to define as specifically as possible the "Who," "What 
Information " "What Purpose," and "How" for its outcome study. Teams were 
challenged to come up with specific indicators of the kinds of "communication 
skills" and other outcomes they wanted the education program to achieve. 

The teams responded well to being encouraged to "be specific," possibly 
because it now gave them something concrete to hold onto as they got into this 
new task of developing their own outcome evaluation. For example, the South 
Cove Manor team agreed that, overall, the study should help determine whether 
the education program was contributing to the nursing home's policy of 
"restorative care." To answer that larger question, information would be 
gathered to answer the following related questions: 

1 . Has there been a decrease in provider-resident conflicts? Has there 
been an increase in positive provider-resident interaction? 

2. Do providers understand and report (in English) changes in resident 
status? (And was appropriate follow-up action taken by the charge 
nurse?) 

3. Are participants using English for an increasing number of work- 
related and non-work-related purposes? (These uses of English 
would be taken from various lists developed by students, the state 
nursing association, and other sources.) 

In the industrial setting of United Electric Controls , the evaluation team decided 
that, overall, the study should help determine whether the education program 
was contributing to the company's goals of introducing innovations like "reliable 
methods," teamwork, and cross-training. To get at that information, the following 
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more-specific questions related to "reliable methods" would be some of the 
focal points of the study: 

1. How well can workers understand job-related documents like yearly 
performance reviews, blueprints, and the X-matrix? 

2. Do workers communicate effectively with ethnic groups other than their 
own? 

3. Do workers work more effectively together in team situations like 
CEDAC groups, action centers, production work cells, and cross- 
training activities? 

At a meeting of the Kennedy Die Castings team on February 21 , team members 
first agreed to focus on a combination of job-related abilities and personal 
goals, the latter of which were identified by the students as important. The team 
then braJnstormed numerous examples for each category of job-related and 
personal outcomes and then decided to narrow down the list by identifying the 
job-skill-areas to pay the most attention to. Some of the resulting "outcome 
questions" were as follows: 

1 . Do students better understand and carry out existing and new 
procedures in the plant, including: (* • highest priority) 

- Sequencing* 

- Trouble-shooting/problem-solving (understanding causes of 
defects/problems and taking needed action)* 

-- Running the machines*; ability to operate all machines 

- Communicating with supervisors and fellow employees* 

- Safety procedures (both machine-specific* and plant-wide) 

- Quality assurance procedures (SPC, quality checks...) 

- General shop rules, structure, benefits (covered in employee 
handbook) 

- Math-related tasks (for production cards/tags) 

2. Are students better prepared to: 
-- Get new/other jobs 
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- Go on to further education and training (inside or outside the plant) 

3. Have students acquired personal skills needed to perform tasks 
related to: 

- Income tax 

- Citizenship/U.S. history 

- Health care 

- Paying bills 1 

In addition to providing focal points for the outcome study, the above process of 
clarifying the intended outcomes of the education program was seen by the 
education providers as a model of a goal-setting process which had other uses 
at Kennedy and at the other worksites where the education provider operated. 
Specifically, a similar goal-setting process was seen as a useful way of getting 
all players in the program to give input and thereby buy into the program. At 
Kennedy, the program's instructor took this process a few steps further by, first, 
meeting for an hour with students to ask them to develop their own list of 
outcomes to focus on in the education program. The instructor then met with 
supervisors and human resources staff to get them to develop their own set of 
program goals. The various lists of "outcomes" or "goals" produced in these 
goal-setting sessions with the evaluation team, the students, and supervisors 
and human resource staff were then synthesized and became the basis not only 
for the data-gathering instruments to be used in this outcome study but for a 
refinement of the curriculum itself. 

The team at Boston CRv Hospital likewise chose to focus on a combination of 
job-related and personal-development goals. Union-related outcome questions 
were also identified. Examples included: 

Job-related outcomes: 

1. Does the nurses' aide participate more fully in staff meetings by: 

- Speaking about the topic 

- Appearing "engaged" 

- Giving concrete suggestions for improved patient care? 



1 These activities were supported directly by the employer. 
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2. Does the aide appropriately use work-related documents by, for 
example, demonstrating an understanding of the content of forms used 
on the job? 

3. Does the aide communicate appropriately with supervisors: 
-- When dealing with emotionally-charged issues 

- In meetings between supervisors and aides 

- Using conflict-resolution skills? 

Union-related outcomes: 

1 . Does the aide read - and contribute to - the union bulletin? 

2. Is the employee more willing to run for shop steward? 

Personal-development outcomes: 

Has the student improved his/her academic skills and gone on to further 
academic training, job-training, or job-promotion, as evidenced by: 

- Has the employee earned a GED? 

-- Has the employee applied for higher education or training programs, 
including: 

• Technical training programs for para-professionals; 

• Pre-tech classes (beyond ABE) 

• College 

- Has the employee been promoted (either within BCH or outside) as a 
result of skills (e.g., reading, writing, oral English) gained in the program? 

Thus, by early March, all four of these teams had come up with a working set of 
outcomes to focus on. 

In response to the question of "What Purpose" these evaluations would serve, a 
common purpose cited by all four teams was to know whether the investments 
made in the program were worthwhile. In turn, the teams could use this 
information to strengthen their proposals to funding sources while at the same 
time clarifying how to improve the program. 
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In terms of ■How" this information would be gathered, all four teams agreed that 
some of this information could be gathered through interviews and/or 
questionnaires with supervisory personnel and with students themselves. The 
teams would also consider using simulations, reviews of documents, and -- in 
the case of South Cove Manor - interviews with residents and a simulated flow- 
sheet exercise. It was agreed that the next step in the process would be for the 
consultant to draft samples of these questionnaires, interview guides, and other 
instruments and then send them back to the teams for revision. 

As facilitator of these initial team meetings, the consultant recorded the teams* 
responses on flip-charts and subsequently sent summaries of the discussions to 
the respective teams. These summaries in effect served as statements of 
commonly-agreed-upon purposes, focal points, and data-gathering activities for 
each team's outcome study. 

It was at these initial site visits that the consultant also took the opportunity to 
take a tour of the worksites conducted by key members of the respective teams. 

Step 5: Teams agree on details of the data-gathering activities to be used: 
April - May, 1991 

As in Step 4, the two consultants followed essentially the same approach to 
helping their respective teams finalize details of how data would be gathered. 
For comparative purposes, the two consultants' activities are described 
separately, as follows: 

Second Mee tinn with Hampden Paper and Aerovox 
. Preparing fnr the Second Meeting 

As the consultant reviewed the data gathered during the first site visits to both 
Hampden Paper and Aerovox, a similarity between the companies and their 
respective education programs emerged. Although the two companies 
implemented their vision in different ways, they both saw education as a Jcng 
term strategy for helping workers adapt to changing production methods. They 
were both on the path to becoming the "high productivity" workplaces which 
produce customized, high-quality product on high-tech machinery with a flexible 
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workforce. This similarity guided the development of the data-gathering 
instruments at these sites. 

In its 1988 report, "Workplace Basics: the Skills Employers Want," the American 
Society for Training and Development (ASTD) suggested that there are seven 
groups of skills in which workers in the high productivity workplace must excel. 
They underscore that effective workplace education is more than the traditional 
three R's adapted to the workplace, and also more than narrowly-defined job- 
specific training. The skill groups which workplace education should now focus 
on are: 

(1) foundation -knowing how to leam 

(2) communication - listening and oral communication 

(3) adaptability - creative thinking and problem solving 

(4) personal management - self esteem, goal setting, motivation, and 
personal and career development 

(5) group effectiveness - interpersonal skills, negotiation, and teamwork 

(6) influence - organizational effectiveness and leadership 

(7) competence - reading, writing, and computation 

The evaluation teams at both sites had agreed that these very skills were 
essential components of the change their programs want to achieve. But it was 
not clear how to define or measure these skills. Between the first and second 
site visits, the consultant had several conversations with the program 
coordinators about the importance of capturing these outcomes in addition to 
the job-specific outcomes which the teams had identified. As a result of these 
conversations, the consultant de veloped a draft questionnaire using the ASTD 
skills areas as a framework, and adapted the draft to each company's needs. 

The consultant identified three or four behaviors for each of the "foundation," 
"communication," "adaptability," "personal management ," "group effectiveness," 
and "influence" skill groups. The following examples of behaviors which 
correspond to the ASTD skill areas are taken from the draft "worker self-reports" 
which the consultant submitted to the evaluation teams at Hampden Paper and 
Aerovox. (The phrasing differs for the supervisors' reports where a supervisor is 
asked to comment on someone else's behavior.) 



• "Foundation* was broken down into: "I like to team," "I am not afraid to ask 
questions about something I donl know," and "I know where to go for 
information to answer my questions (to a book, a person, etc.)." 

• "Communication" was broken down into: "I understand directions;" "I listen to 
my co-workers' questions;" "I listen to my family members: spouse, children, 
other family, friends;" "I ask questions clearly;" and "I need less translation 
from my native language into English." 

• "Adaptability* was demonstrated by: "I donl wait for someone else to tell me 
what to do. I figure it out for myself;" "I think about new/better ways to do things 
at work;" and "I discover and apply new/better ways to do things at work." 

• " Personal management* was broken down Into: "I value my opinions and 
contributions;" "I offer my opinions to others;" "I plan ahead;" "I am comfortable 
working alone;" and "I am a more active member of my community - my union, 
church, community center, citizenship group, other." 

• "Group effectiveness " was demonstrated by: "I work well with my co-workers;" 
"I have a sense of team work and team identity;" "I negotiate my needs and the 
needs of my group with my supervisor," "I am supportive of others' efforts to 
increase their skills;" and "I am supportive of others' learning efforts at home 
my spouse, children, other family, friends." 

• "influence" was demonstrated by: "I understand the importance of quality 
awareness in the company's strategy to improve production and sales;" "I 
understand what SPC is;" "I understand the role that SPC plays in the 
company's strategy to improve production and sales;" "I understand the 
relationship between the parts of the work process I perform and the whole 
production process;" "I understand the long term goals of (company name) 
and the essential role that education plays in making these ^oals a reality;" 
and "I understand that it is possible for me to advance in the company as I 
increase my skills." 

• "Competency" was broken down into the reading, writing, and math 
competencies that the teachers define for each of the classes they teach. 
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(This means that competencies differ for ESL, ABE, SPC math, and computer 
training courses.) 

Thus, this second set of outcomes was added to the job-specific outcomes 
which the teams had identified as important to measure during their first 
evaluation team meetings. This second set of outcomes came to be known 
simply as the "ASTD categories." Although the competencies are technically a 
part of the ASTD framework, in the cases of Hampden Paper and Aerovox, they 
formed a third section of the sample questionnaires. 

Thus, the sample questionnaires which the consultant drafted and submitted to 
the evaluation team members at Hampden Paper and Aerovox for review prior 
to the second team meetings consisted of three parts: (1 ) the job specific skills 
identified in the first meeting; (2) the "ASTD categories" described above; and 
(3) competencies in reading, writing, and math defined class-by-class. 
The instruments were designed to collect information on similarly-defined 
variables in similar formats from different respondents, thereby providing a 
system of checks and balances on the data. Workers and supervisors would be 
the main respondent groups for the variables listed above in both companies. 

In addition, Hampden Paper is unionized and union leadership made it clear in 
the first team meeting that it was interested in understanding the impact of the 
education program on union-related issues. As a result, the consultant drafted 
a union checklist which assesses worker perception of program impact on: 

- Awareness of the range of benefits offered through the union 

- Identifying problems on the shop floor as union concerns 

- Understanding the grievance procedure 

- interest in becoming a steward or union officer 

- Desire to negotiate benefits on union contracts 
Understanding the union role in creating and supporting your 
workplace education program 

- Understanding the terms of your union contract (as reflected in the 
curriculum or from other sources) 



Team members were asked to revise these drafts in preparation for the second 
meeting. 

* Goals of the Second Meeting 

The goals of the second meeting were to: 

(1) finalize the outcomes to be measured; 

(2) agree on the data-collection formats; 

(3) designate who will collect the data; 

(4) decide who will analyze and report the data under what time line; 

(5) clarify how the data will be used (how can it lead to some useful 
action) 

* Results of the Second Meeting 

The teams at Hampden Paper and Aerovox achieved similar goals in their 
second meetings. These include: 

1. The teams finalized their list of outcomes: 

At Hampden Paper the team agreed to put aside its initial interest in 
enrollment, attendance, and promotion as indicators of program success. 
These indicators are more linked with the new, soon-to-be-implemented 
certification program than they are with the current program. These 
outcomes can be explored once the new program is underway. At Aerovox, 
the coordinator, with input from all the team members, presented a working 
list of job-specific outcomes which would become the basis of the worker 
and department head questionnaires. 

2. The teams agreed on a questionnaire format: 

- The questionnaire format in both programs was limited to multiple-choice 
questionnaires, supplemented by a few open-ended questions for 
supervisors. Although the language varied for the two programs, the 
questions essentially asked whether the respondents had "stayed the 
same; "improved a little," or "improved a lot." This format permitted the 
greatest amount of data collection in the least amount of time. The 
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amount of time required to collect data was a real concern for both teams, 
especially the amount of time required of the supervisors. Students still 
enrolled in a program could be asked to complete a questionnaire as part 
of a class session. Students not currently enrolled posed a problem 
similar to the supervisors - how much time could the evaluation project 
legitimately ask of them? 

-- Both teams agreed to have: 

(1 ) a student self-report to be completed by as many students as possible, 
those formerly and currently enrolled; 

(2) a supervisor questionnaire to be completed by any supervisors or 
department heads having students in their charge. The teams agreed that 
the supervisor questionnaire would cover essentially the same areas as 
the student self-report but would not include questions on educational 
competencies or on areas of learning that impact a student's life outside of 
work, on family or community. It would, however, include additional 
questions on suprvisors' attitudes toward the education program in 
general. Both teams agreed that the supervisors would respond in the 
aggregate and not for each individual in their charge. It was decided that 
responding for each individual in their charge would be too demanding at 
this time. The process of develc , g the supervisor questionnaires was 
genuinely collaborative. A supervisor at Hampden Paper took great 
interest in the supervisor checklist , and revised it substantially before 
submitting it to the team for review. Similarly, at Aerovox, the coordinator 
shortened the questionnaire for department heads with input from team 
members. The coordinator at Aerovox also had the added benefit of 
reviewing the Hampden Paper revised supervisor questionnaire before 
finalizing the Aerovox questionnaire. Thus, in this phase of the 
evaluation, there was collaboration between programs as well as within 
them. 

(3) at Hampden Paper all workers would also complete the union 
checklist. 

3. The teams agreed to try to generate statistical data: 



it was agreed at this time that the kind of data that the two programs would 
be generating and the amount of it - from an estimated eighty students and 
eight department heads at Aerovox, and twenty students and seven 
supervisors at Hampden Paper - would make statistical analysis a 
particularly suitable option. Such analysis could produce useful information 
through cross tabulation by student age, gender, department, job title, 
language spoken at home, ethnicity, time in company, time in U.S., 
education history, and reasons for entering a workplace education program. 
The teams agreed that basic statistical data - frequencies, cross tabulations, 
and simple tests of significance - would make the data more appealing in 
the eyes of potential funders, whether the funder is the company itself or an 
outside agency. The computer training described in Step 7 below prepared 
the program coordinators to think about and work toward long-term 
systematic data collection and analysis. 

The coordinators took on the main responsibility for collecting, programming, 
and analyzing the data, with help at Hampden Paper from a student in the 
program who is particularly interested in computers. Data collection and 
analysis were scheduled to be completed by mid to late summer. 

4. The teams discussed how the findings would be reported: 

It was not clear how the data could/would be reported or used untii they 
were available for analysis. Both teams were concerned about 
confidentiality and agreed to code their protocols so that only the coordinator 
would know the identity of the respondents. 

Second Meeting with Kennedy Die Cast ings. South Cove Manor. Boston City 
Hospital, and United Electric Controls 

In the month that followed his first visit to the above four programs, the 
consultant drew up drafts of the questionnaires, interview guides, and 
simulations which the four teams had agreed to develop. These drafts were 
circulated to team members for their review, and then during the period of April 
9th to 1 1th each team met again to decide how these and other data-gathering 
instruments would be used. 
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In these meetings, the consultant again served as facilitator of two-hour 
meetings for each group. Team members were walked through a series of 
questions aimed at getting the teams to think through and commit themselves to 
actions which the data-gathering process would require. Those questions and 
discussions were as follows: 

Question #1: Which data-oatharing activities will we use? 

In most cases, the teams agreed to gather data through a combination of 
questionnaires (or interviews) aimed at participating workers and their 
supervisors. These instruments were selected in pan because it was felt that 
they - in contrast to observation of workers on the job or other data-gathering 
techniques - could elicit data without requiring a large amount of time of either 
the informants or the data gatherers. 

South Cove Manor also decided to use classroom simulations in which 
students would be posed with a problem situation taken from their jobs and then 
asked to demonstrate how they would solve that problem. Although this method 
was more time-consuming for those involved, it was felt that it would elicit richer 
data while also, in effect, serving as an educational activity for students. South 
Cove Manor also decided to develop a flow sheet exercise in which students 
would be asked to complete a sample flow sheet to demonstrate their ability to 
record patient status on a particular day. And the South Cove team also agreed 
to interview selected English-speaking residents at the nursing home, to ask 
them to assess how well the ESL students use English in their interactions with 
the residents. At United Electric Controls, the human resources officer said he 
might want to look at the "Valued Ideas" suggestions submitted by workers to 
see if there had been any increase in the quantity or quality of ideas submitted 
by ESL students. (This data-gathering method was later shelved by the UEC 
team as priority was given to questionnaires for students and supervisors.) 

Question #2: For each data-gathering activity, wha t will be the content and 

format? 

Each team reviewed the drafts of questionnaires, interview guides, and 
simulations which the consultant had circulated to team members prior to this 



meeting. Team members now took the opportunity to make suggestions - in 
some cases minor changes and in some cases major revisions about what 
the questions should focus on (that is, the "content") and how the gathering of 
the data should be organized or "formatted" (for example, layout of 
questionnaires, steps for the simulations). 

The content generally focused on the kinds of outcomes identified in Step 4 
above. However, as the teams got further into the work of designing data- 
gathering instruments, they began to realize that their time was limited and they 
would have to limit the amount of data to be collected. For this reason, the 
teams tended to narrow down their focus to Job-related outcomes rather than 
outcomes related more to learners' personal development 

Question *3: Hnw will the data be used? 

This question aimed at getting the teams to decide on such issues as 
confidentiality, whether the results would be reported in aggregate or by 
individual, and whether data would be categorized in any special way (such as 
how long the student had been in the program, or the students' place of origin, 
educational background, or job category). 

On the question of confidentiality, all teams agreed that the identity of students 
would be kept confidential. Only the program coordinators and teachers would 
know which student received which score. This was done with full approval of 
the company managers who said that they just wanted to know how well the 
program was working rather than know how any one student was doing. It was 
agreed that this would encourage the students and supervisors involved in the 
study to be more frank in their answers. The teams agreed to make this clear 
when explaining the study to the participants. 

It was also agreed that, by allowing the relatively "neutral" coordinators and 
instructors to know the students' scores, the education staff could counsel the 
students and revise the curricula to help meet student needs identified in the 
data. This arrangement would also allow the education staff to track student 
performance over time. 



In terms of categorizing results by students' Job, background, and so forth, most 
teams agreed to merely consider the question and wait until the data were 
gathered before deciding whether any particular categorizations would be 
useful. One team said that it wanted to avoid any hint of ethnic or racial 
discrimination and thus would avoid categorizing results by ethnic criteria. At 
South Cove Manor, the team thought that it might be useful to compare the 
performance of, say, mainland Chinese workers and Hong Kong- or Taiwan- 
bom employees. SCM said it might also consider comparing the results of 
students with limited formal education with those of more-highly-educated 
workers. 

Question #4: How will the data be collected? 

The teams generally agreed to now work with the consultant to do last-minute 
revisions to the drafts of the agreed-upon instruments before field-testing them 
with a sample of informants. If necessary - and if time permitted - the teams 
would then again fine-tune the instruments and then use the instruments to 
gather the first round of data by approximately mid-May. 

Once the data were in hand, the teams would circulate the data among team 
members to allow them to study the data before analysis and reporting of the 
data were carried out. 

Question #5: How will the data be analyzed and reported? 

The teams generally agreed to leave the "dirty work" of tabulating and 
summarizing the raw data in the hands of one or two key people on the team, 
usually the coordinator and/or instructor. The teams seemed to feel that these 
people had the most time to devote to the process and perhaps the best day-to- 
day knowledge of both the students and the process of assessing their skills. 

The coordinators seemed willing to take on this extra job because they saw 
value in producing a clearer picture of what their programs were accomplishing. 
They were often concerned, however, about the additional time that would be 
required of them. They also expressed some concern that they might lack the 
technical skills required to properly interpret and present the data. 
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It was in response to this question about the coordinators' technical skills that 
the consultants organized a special training workshop for four of the 
coordinators. (See Step 7, "Special Training Provided on Data Analysis," 
below.) 

At this point it was agreed that, until the data were actually in hand, it would be 
difficult to be certain about how the data would be analyzed and reported. The 
teams thus agreed to wait until after the data were collected to decide exactly 
how the information would be used. 

Question M: What resources need to be committor! and under what time-line? 

The teams agreed that the primary resources required for the above data- 
collection and analysis would be the staff time of those involved and some 
additional computer and clerical resources. It was hoped that the first round of 
data could be gathered by mid-May and then analyzed by mid-June. 
Preparation of reports would come by the end of June. 

Step 6: Teams gather data: May - September 1991 

The actual gathering of data in the six programs went more or less according to 
the schedule agreed on above. However, some of this data-collection was 
complicated by unforeseen staff changes and interruptions of work schedules. 
Specifically, the coordinator of the Boston City Hospital program left his job 
before the data could be collected, leaving that task in the hands of other team 
members who had to quickly organize themselves to take on the responsibility 
of distributing, gathering, and analyzing questionnaires. These BCH team 
members also had to deal with the fact that the hospital was laying off and 
transferring workers and supervisors, thus making the task of tracking down 
these informants and distributing the questionnaires to them that much more 
difficult. 

Similar temporary layoffs and schedule changes around this time affected a 
number of other sites. The primary result of these disruptions was a reduction in 
the numbers of questionnaires distributed and, thus, a reduction in the 
representativeness of the data actually gathered. Closing down or slowing 
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down some programs for summer, the scheduling of the computer training in 
June, and workers taking vacations also slowed down the data-gathering 
process. 

The following describes the data-collection processes at each of the sites. 
Hampden Paper 

The three questionnaires which the Hampden Paper evaluation team agreed to 
use in its outcome study were ready in late June, after classes ended for the 
summer. The three questionnaires were: (1) the student self report (three 
slightly-different versions of which were developed to make them relevant to the 
three classes which Hampden Paper offers - combined ESUABE; GED; and 
computer training); (2) the supervisor report ; and (3) the union member report. 
All students are union members. Those who completed a self report also 
completed a union report . The content of these questionnaires is described in 
Steps 4 & 5, above. 

The coordinator asked individual students to complete their questionnaires 
when she sought them out at their work stations, or when she saw them 
elsewhere in the plant. Students completed the questionnaires in the 
classroom at their convenience, with the help of the teacher/coordinator if they 
needed it. Data collection with the students continued slowly with a total of 
twenty completing questionnaires. Supervisors were given their questionnaires 
by the supervisor member of the evaluation team. Responses were received 
from all seven supervisors who had students in their departments. The student 
self-report covered three areas: job- specific skills; "ASTD categories", and 
educational competencies. The supervisor questionnaire covered job-specific 
skills and other areas related to supervisors' understanding of and support for 
their education programs. 

A student in the computer class at Hampden Paper who was not a member of 
the evaluation team helped the coordinator develop the computer program on a 
Hampden Paper database which would be used to analyze the evaluation data. 
He also punched in and analyzed the data with the coordinator. This was done 
with the full support of the company president who encouraged worker 



participation in the evaluation project, especially because it was seen as 
developing skills which are relevant to the workplace. Involving students more 
fully in evaluation activities like this was seen as one way to take the burden of 
the evaluation work off the coordinator and build worker investment in the 
program. This student had military duty for two weeks in July, however, and this 
contributed to a slow-down of the data-gathering process. Data were collected 
until the middle of August. 

Aercvox 

The Aerovox outcome evaluation consisted of two questionnaires - the student 
self-report questionnaire and the department head questionnaire. The student 
self-report varied slightly for students enrolled in different classes: ESL, ABE, 
GED, or SPC math. The student self-report covered three areas: job- specific 
skills; "ASTD categories," and educational competencies. The department 
head questionnaire covered job-specific skills and other areas related to 
department heads' understanding of and support for their education programs. 
The content of these questionnaires is described in Steps 4 & 5, above. 

The education program shut down, along with the company, for the first two 
weeks in July. When workers and program staff returned, the coordinator 
finalized the questionnaires and soon began data collection on students who 
had already completed a class - about sixty-five people. She waited until the 
end of August to collect data from another twelve workers currently enrolled in a 
class so they would have the benefit of a full class cycle before they responded. 
Students who had completed a class filled out their self-reports at their 
convenience in the classroom with the coordinator's help if they needed it. 
Students currently enrolled in class filled out their self-reports at the end of a 
class period. Nine department heads were given their questionnaires at the 
same time as the enrolled students. Department heads were to fill out their 
questionnaires at their convenience. As of mid-September only two department 
heads had completed their questionnaires, most likely because the company 
was under extreme pressure to keep up with production while the workforce 
was cut back to a four-day work week. With some prodding by the program 
coordinator, and additional explanation of why the evaluation was important, 
the department heads finally completed their questionnaires. 
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South Cove Manor 

In May, the South Cove Manor team gathered its data in the following activities: 

~ Questionnaires for English-sneaking supervisors and nurses : 
Supervisors and licensed staff who had to rely on English as their means 
of communicating with other employees answered sixteen questions 
about their interactions with students. The questions focused on the 
communications involved in students' reporting of changes in patient 
condition and supervisors' responses to those reports. Eight supervisory 
staff completed evaluations for twelve students, with each student being 
assessed by at least two different supervisory personnel. 

- Self-assessment questionnaires for students: students completed a 
Chinese-language questionnaire in which they rated their own abilities to 
perform sixteen tasks required for effective interaction with residents in 
English. They were also asked to describe areas where 
misunderstandings occur between them and English-speaking residents. 
A third set of questions focused on how well the students can report 
changes in patient condition to their English-speaking supervisors and 
charge nurses as well as how well these supervisory staff respond to the 
students' reports. Students also assessed how much the ESL program 
had helped them accomplish personal goals like "making new friends," 
"understanding American culture," and "getting more education." 
Twenty-three students completed these questionnaires. 

- interviews with residents : Eight English-speaking residents were 
interviewed about their interactions with the ESL students. This interview 
was a short one, with only a half-dozen questions. Residents were 
asked, for example, whether a particular student speaks English when 
helping with bathing, dressing, and other daily activities. Twelve 
students were assessed through these interviews. 

- Simulations : Nineteen of twent-two students participated in classroom 
simulations organized as follows: First, instructors explained the 
situation in Chinese, telling the student to perform a simulated task 
requiring the use of English. (For example, in one case the student was 
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told to interact with an English-speaking patient while shaving him and 
then report the patient's status to the charge nurse.) In this case, the 
"patient" (played by a staff member) meanwhile had his own instructions 
to say things in English about "a skin rash" and other issues which the 
student had to respond to. The student was rated on his/her ability to 
perform 30 different tasks on a scale of "1" (unacceptable), "2" 
(incomplete), or "3" (acceptable). 

- Flow sheet exercise : Twenty-one students were presented with a 
typical situation in which they were to help an elderly patient groom 
herself, use the bathroom, and take a walk. The students were then 
asked to fill out - in written English - a daily flow sheet for that patient, 
using a code to describe what the patient did and how well she did it. 
The students were then rated on how accurately and clearly they 
completed their respective flow sheets. 

Kennedy Die Castings 

At Kennedy Die Castings, data were gathered from one supervisor and ten 
students at two points - June and August - in one of the education program 
cycles. 

Due to staffing changes within the company, only one supervisor was available 
to be interviewed, and he twice provided ratings for ten students' abilities to 
perform seven job tasks. (The team decided to use an interview format for the 
supervisor rather than distribute questionnaires, because previous attempts to 
get supervisors to respond to questionnaires about the education program had 
produced a meager response from supervisors.) This time, the supervisor fully 
understood the purpose of the interview and gave it full, careful attention. He in 
fact helped the instructor to field-test two different versions of the interview -- a 
longer version and a shorter one - before it was decided that the shorter one be 
used. In the interview, the supervisor used a scale of 0-7 to rate workers' 
abilities to "interact with work peers," "recognize and identify quality defects," 
"participate in group problem-solving on the floor," "participate in training 
programs," and "comprehend and follow directions." His two sets of responses 
provided data which could be used to identify changes in students' abilities 
between June and August. 



Ten students were twice asked to complete a questionnaire to assess their 
abilities in the same job-task areas covered in the supervisor interview. When 
the questionnaire was first distributed in June, the instructor wanted to assure 
learners of confidentiality and thus told the students not to put their names on 
the questionnaire. When the same questionnaire was distributed in August, 
however, the students said they didn't mind putting their names on the 
document and did so. The data from the two uses of the student self- 
assessments could thus be used for comparative purposes only in aggregate 
form and not on a student-by-student basis . 

The ten students completed their questionnaires during class time. These 
students represented a mix of English-language skill levels. Students were 
encouraged to be honest in their self-assessments. For each skill-area, 
students could choose from "I donl do it at all," "I can do It a little, - "I do it very 
often," or "I do '.l all the time." The instructor encouraged more-advanced- 
English-speakers to work with th?ir less-advanced compatriots to explain terms 
used in the questionnaires. 

The instructor also tried - with the help of the supervisor - to use a simulation 
exercise as another means of gathering data about students* skills. (The 
supervisor even served as a guest instructor in some classes in an attempt to 
prepare the students for the tasks they would be assessed on in the simulation.) 
However, this attempt at using a simulation showed that the complex nature of 
the skills to be assessed was difficult to capture in a simulation, so the 
simulation idea was shelved until a future time when more attention could be 
given to designing a more-meaningful version. (See the Kennedy section 
under Step 8 below for further discussion of this attempt to use a simulation 
activity.) 

United Electric Controls 

In the period of July to September, four United Electric supervisors completed 
questionnaires in which they assessed the abilities of a total of fourteen 
employees. The questions focused on how well students were using English- 
language communication skills to participate in several innovations recently 
introduced by the company to shift decision-making into shop-floor employees' 



hands. Several of the questions dealt with problem-solving team activities tike 
"action centers," "production work ceils," and "CEDAC groups." Other questions 
asked how fully the students participated in cross-training activities, the "Valued 
Ideas Program" (in which all employees are encouraged to submit suggestions 
for improving the workplace), and "ergonomics" (a system for making best use 
of workers' body mechanics). The supervisors checked how well the student 
performed the given tasks, with choices ranging from "Employee cant do this at 
all." to "Employee does this well." 

Nineteen students were able to complete self-assessment questionnaires, with 
an additional student unable to do so due to limited literacy skills. The 
questionnaires focused on essentially the same job-related skill areas covered 
in the supervisor's questionnaire. Students chose from answers ranging from "I 
can't do this at ait." to "I do it welt." 

Boston City Hospital 

Due to personnel changes occurring throughout the Boston City Hospital 
system at the time the data were to be collected, and because the program 
coordinator left at the same time, it was difficult for the evaluation team to pull 
together the students and supervisors for whom questionnaires had been 
designed. 

Despite these logistical limitations, a total of seven students completed self- 
assessment questionnaires. They assessed how well they could perform a 
dozen or so job tasks like "handling emergencies and conflicts," 
"communicating with supervisors," "filling out records," and "anticipating needs 
and taking initiative." They also assessed how well they could deal with three 
union-related tasks: "reading the union bulletin," "writing an article for the 
bulletin," and "speaking about the education program at union meetings." They 
also described where they currently stood in terms of achieving personal goals 
like pursuing further education, training, and job-promotions. 

A total of six supervisors completed questionnaires assessing the skills of six 
different students. These supervisors assessed the abilities of students to 
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handle essentially the same dozen job tasks covered in the student 
questionnaire. 

Step 7: Special training provided on data analysis: June 1991 

In April, the coordinator of the South Cove Manor program asked whether the 
consultants could in some way help him develop a computerized system for 
tabulating and analyzing the data his team was about to gather. In response to 
this request, and in response to the emerging need for computerized analysis of 
the data at Hampden Paper and Aerovox, the consultants arranged to have a 
special half-day training course set up for any coordinators interested in 
learning how to use the Statistical Program for the Social Sciences (SPSS) 
computer software. This workshop was conducted by Alice Oberfield, a 
statistics specialist, on June 18, 1991. 

Coordinators from the South Cove Manor, Kennedy Die Castings, Aerovox, and 
Hampden Paper programs participated in this training which was tailored to the 
outcome study needs of the programs represented. Participants were led 
through a series of activities in a four-hour period which familiarized them with 
the uses of the SPSS system, how to code their outcome data and use 
frequencies, cross tabulations, and Chi-Square tests of significance on them. 
They left the workshop with some basic skills they would need to further explore 
and use the SPSS system to analyze the data they were gathering back home. 

Step 8: Data analyzed: June • November 1991 

From June through November, the teams tabulated and analyzed the data they 
had gathered in Step 6. This was done in part with input from the two 
consultants who first reviewed the data which had been gathered and then 
gave feedback to the teams. The consultants provided this feedback via 
telephone conversations with program coordinators and in the third set of on- 
site meetings the consultants held with each evaluation team. (Two teams 
Hampden Paper and Aerovox - required a fourth meeting as well.) The 
following is a description of the data gathered by each team and how it was 
interpreted by the team: 
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Hampden Paper 



By late September, the coordinator at Hampden Paper had a sample of her 
data analyzed on a database software program commonly used in the company 
for a variety of purposes. The coordinator had been advised by the company's 
computer specialist that this database program would readily serve the outcome 
study's need to compute frequencies and cross tabulations. In fact, the program 
turned out to be cumbersome. Given the other option of starting over again from 
scratch on SPSS at this point in time, the coordinator chose to labor with the 
paper-heavy database program, and to complete her analysis of the data by the 
end of October. 

The preliminary results of the Hampden Paper outcome study showed that both 
workers and supervisors observe significant improvement among students on 
selected skills, that workers tend to report higher rates of improvement than 
supervisors in those skill areas, and that participation in the program has 
improved the "labor consciousness" of workers. (Improvement is here noted as 
the total percentage of "changed a little" and "changed a lot." ) The consultant 
met with the coordinator alone during the third on-site visit and helped to 
interpret the preliminary data. The complete results were reported to the 
evaluation team at the fourth and final team meeting. The coordinator then met 
with the company president (who also served as a member of the team) to 
discuss any items of interest that may not have been covered during the fourth 
visit. The coordinator then wrote the final report. 

At Hampden Paper, twenty students and seven supervisors completed 
questionnaires. Nine of the workers are enrolled in the ABE program, five are 
enrolled in the GED program, and six are enrolled in the computer skills course. 
The final data analysis included cross tabulations by class as well as by 
department. The questionnaire which was given to students gave them the 
option to chose as answers: "pre" (that is, had the skill coming into the class); 
improved a little; improved a lot; stayed the same. The findings of the outcome 
study at Hampden Paper are summarized below. (The complete report of the 
program coordinator is contained in Appendix A, under separate cover.) 
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in general, the Hampden Paper students reported consistent and noteworthy 
improvement in job-specific skills and in the ASTD categories, and 
demonstrated a noteworthy increase in "labor consciousness" in the responses 
to the union questionnaire. Supervisors noted change, too, although less of it 
than did the students. The differences between the student and supervisor 
responses in most categories were small enough to see that there is substantial 
agreement about what the program is achieving. Reasons for discrepancies 
between supervisors' and students' responses are discussed in a previous 
section or this report . (See pages 16-18.) But in the Hampden Paper case, 
there is an additional methodological reason for the discrepancies. While 
students had the option to answer "pre", that is, had the skill already coming into 
class, supervisors were not given the same choice. Thus, if a supervisor did not 
note change, it might be because the students already had the skill, not 
because the students did not learn. Taking this into account, the following 
figures, taken from the coordinator's report, for a sample of the most important 
job-specific skills studied, demonstrate more congruence between supervisors' 
and students' responses than not. The sorting of the data by department and 
class provides insights into the effectiveness of the program. 

It is interesting to note that, in her analyses, the program coordinator chose to 
include some variables from the ASTD category of skills in the job-specific 
category. This indicates a change in thinking about what job-specific means. 

Job-Specific Skills : 

- Understanding directions : 42.9% of supervisors noted improvement in 
students' understanding directions. 71.4% of ABE students noted 
improvement while only 1 GEO and 1 computer student noted any 
improvement, the others showing a prior skill. It is to be expected that 
GED and computer students will have better communication skills than 
the ABE students, a group which includes ESL students. 

- Reading production tickets: 60% of responding supervisors noted 
some improvement in accuracy. 87.5% of ABE students who did not 
already have this skill coming into the class noted improvement. (62.5% 
noted a lot of improvement) This is an important outcome because 
reading production tickets accurately is an essential part of doing the job 
right. 
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- Writing accurate reports : 28% of all supervisors noted some 
improvement. 66.6% of ABE students noted improvement, with 44.4% 
noting "a lot" of improvement. None of the students indicated "pre." 
These data indicate that writing accurate reports and reading production 
tickets are skills that students believe they need to work on, despite the 
fact that ail students have said, during conferences with the teacher, that 
they have no trouble with production tickets. 

- Communicating needs to supervisors: 50% Of responding supervisors 
noted some improvement, with 1, or 16.6% noting "a lot" of improvement. 
This is the only time a supervisor answered "a tot" in this survey. 85.7% 
of all students noted improvement, 28.6% " a tot" and 66.6% "a little." 
ABE students noted no prior skill, whereas 40% of GED students and 
66.6% of computer students indicated a prior skill. This is 
understandable as the self-confidence of people in these latter groups is 
likely to be higher. 

- Asking questions clearlv : 40% of the supervisors noted improvement, 
as did 75% of all students who did not already have this skill coming into 
the class. 

- Offering opinions and ideas : 28.6% of supervisors and 85.7% of 
students noted improvement. Only 33.3% of the students said they 
improved "a lor- lower than for other categories. "Pre" responses were 
low, but increased with skill level. This would seem to be an area to 
focus on with all students in the program. 

-- Working well with others : 28.6% of supervisors noted improvement. 
83.3% of students who did not already have this skill coming into the 
class noted improvement. 

Note: the Receiving/Power Wind Department noted a large improvement 
in the way worksr- communicate their needs on the job. Receiving/ 
Power Wind also noted the greatest overall improvement of any 
department. The quest'on of why this department notes such 
improvement as compared to others remains to be answered. 

ASTD Categories : 

-- Like to learn : 85.7% of students who did not already have this skill 
coming into the class noted improvement. 
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-- Know where to get information : 85.7% of students who did not already 
have this skill coming into the class indicated improvement. This is to be 
expected with the independent learning environment which has been 
created in the classroom. 

-- Value your opinions and contributions : 91 .6% of students who did not 
already have this skill coming into the class noted improvement 
-- Try better ways to do things at work: 92.3% of all students noted 
improvement, 61.5% "a lot." This, again, can be understood considering 
the independence fostered in the classroom. This has an effect on self- 
esteem and initiative. 

- Support family's and friends' efforts to learn: 81.3% of students who did 
not already have this skill coming into the class noted improvement. 

~ Understand long-term ooals of Hampden Paoer: 87.5% of students 
noted improvement. 

Union Issues: 

(The answer-options on this questionnaire were: "decreased, " 
"increased," "stayed the same.") 

- Awareness of union benefits : 55.% of the union-member students 
noted improvement. This increase was the same in most departments, 
except for Cobr/Coating where it was slightly higher. 

- Understanding the terms of the uni on contract : 81 .8% of students who 
did not already have this skill coming into the class noted increased 
understanding. 

- Interest in becoming a union officer: 20% of the total responding to this 
question said that their interest decreased - perhaps because of the 
constraints that participating In the education program puts on them. 
10% reported an increase. 

- Desire to negotiate benefits on uni on contracts : 50% had an increased 
interest. This included 100% of the Shipping/Embossing Department, 
57% of the Fancy Department, and 66% of computer students. This 
would perhaps be higher, at the end of a contract period - like now -- 
when workers are anticipating new negotiations. 

The evaluation team identified scrap reduction as a desirable outcome of their 
workplace education program, but there was no way to track scrap 
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production/reduction this year. As the company continues to install 
computerized equipment, systems for measuring scrap will also be developed. 
This means that tracking scrap production may be an important part of an 
ongoing outcome evaluation at Hampden Paper. 

Aerovox 

By late September, the coordinator at Aerovox had a small sample of responses 
from twelve students who had just completed an ABE class. Data from forty- 
three other students was collected and analyzed by mid-November, bringing 
the total number of students to fifty-five. (Eighty-three questionnaires were 
distributed, and fifty five were returned completed and judged to be useable.) 
Data from nine department heads was also collected by late November. The 
lag between September to November is chiefly the result of the amount of time it 
took to simply collect the data, coupled with the difficulty of mastering the SPSS 
program, and the fact that the coordinator took a vacation during this time. The 
coordinator also decided fairly early on that she would solicit help from a 
student at the University of Massachusetts in D-tmouth, host institution to the 
education staff, to input the data and run the SPSS program. This university 
student was not available until the end of September, and then became ill, 
leaving much of the work to the program coordinator. 

Students were drawn from five different classes. The classes and numbers of 
students from each class are as follows: 

1 . English for Special Purposes (ESL classes with emphasis on job related 
language). Responses from twenty students. 

2. Adult Education 1 (Adult Basic Education). Responses from twelve students. 

3. Adult Education 2 (Pre- Adult Secondary Education and Adult Secondary 
Education). Responses from five students. 

4. GED. Responses from nine students. 

5. Pre-SPC (Statistical Process Control ) Math workshops. Responses from 9 
students. 

Students completed the Work-Related Skills questionnaire, the ASTD 
questionnaire, and the Competencies questionnaire (reading writing and 
computation, especially for use by teachers and program coordinator.) 



Department heads completed a Work-Related Skills questionnaire and 
commented on their attitudes about the literacy program. The department head 
Work-Related Skills responses were cross tabulated with worker/student 
responses and by department. Student responses were cross tabulated by 
department and classes. 

The coordinator considers the responses from the twelve ABE students to be 
the "cleanest" of her data. This is because the students who answered these 
questionnaires were still in class when they old so, and because these students 
completed the questionnaires with the help of a teacher (If they needed it) in a 
classroom at the end of a class period. The chances of these students taking 
the questions lightly, misunderstanding them, or exaggerating their answers 
seemed less than for other groups who had not been t.i the classroom for four 
months. In general, the data on job-related tasks for these twelve students are 
very positive. Students report significant improvement on all the tasks, 
especially ones related to quality awareness, SPC, reading graphs, and 
following their manufacturing process instructions (MPls). One safety-related 
result deserves special attention. 

This summary report of the Aerovox outcome evaluation takes the form of a list 
of job-related skills with percentage of reported improvement by the twelve ABE 
students, followed by percentage of reported improvement by the tola] worker 
group of fifty-five, and by the nine department heads. (Improvement is here 
noted as the total percentage of "changed a little" and "changed a lot.." In the 
coordinator's report , percentages for "changed a little" and "changed a lot" are 
reported separately. The complete report of the program coordinator is 
contained in Appendix A, under separate cover.) 

It is interesting to note that the percentage of reported improvement by the 
twelve students does not vary significantly from the data for the total worker 
group, except in the case of learning about manufacturing process instructions 
(MPls). The twelve students focused a lot of attention on MPls in their ABE 
class. Their self reports of significant improvement are consonant with the 
amount of time spent on MPls in class. This is not the case for the total student 
group where, for example, students in the SPC math class did not focus on 



MPIs. Therefore, the lower percentage of improvement on MPls for the total 
student group reflects the fact that not ail students focused on MPl's in class. 

Job Specific Skills; 

- Can describe my job. 77% of the ABE students noted improvement.; 
83% of the total noted improvement; and 66.8% of the department heads 
noted improvement. 

- Can relate mvtoh to the o verall production of capacitors. 62% Of the 
ABE students noted improvement.; 87% of the total noted improvement; 
and 22.2% of the department heads noted improvement. Obviously, the 
students report understanding something that the supervisors do not see. 
(Reasons for discrepancies between supervisors' and students' 
responses are discussed in a previous section of this report. See pages 
16-18.) 

- I nspect parts of my equipment. 54% of the ABE students noted 
improvement; 65.5% of the total noted improvement; and 55.6% of the 
department heads noted improvement. Some students come into class 
already knowing how to inspect their equipment, so the percentage of 
students reporting improvement is somewhat lower for this task than for 
others. 

- Practice maintenance of mv equipment. 61 % of the ABE students 
noted improvement; 67.3% of the total noted improvement; and 44.4% of 
the department heads noted improvement. Some students come into 
class already knowing how to maintain their equipment, so the 
percentage of students reporting improvement is somewhat tower for this 
task than for others. 

- Practice cleanliness of my equipment/surroundings . 76% of the ABE 
students noted improvement; 76.3% of the total noted improvement; and 
22.2% of the department heads noted improvement. 

-- Take safety precautions. 85% of the ABE students noted improvement; 
81 .9% of the total noted improvement; and f>e % of the department 
heads noted improvement. The number of vork. «"s compensation 
claims has been reduced so significantly in the last year that the Vice 
President for Human Resources is considering a change in the 
company's insurance policy that could save the company around 



$150,000 next year. (See page 19, and the coordinator's report in 
Appendix A.) 

- Understand the need for Quality Control, 93% of the abe students 

noted improvement; 87% of the total noted improvement; and 44.4% of 
the department heads noted improvement. From the perspective of 
management, quality control is the heart of what workers need to learn in 
their workplace education classes. Aero vox can compete with its major 
competitor only on the basis of superior product and short delivery time. 
A 93% improvement response is especially heartening to management. 

- Understand the purpose of having an MPI. 77% of the ABE students 
noted improvement; 56.3% of the total noted improvement; and 55.6% of 
the department heads noted improvement. 

Can follow my MPI. 85% of the ABE students noted improvement; 
49.1% of the total noted improvement; and 55.6% of the department 
heads noted improvement. This is one area where supervisors note 
more improvement than the total group. It is likely that they are 
responding to the dramatic improvement reported by - and presumably 
demonstrated on the floor by - a smaller number of students. 

- Can follow MPIs from oth er sections . 85% of the ABE students noted 
improvement; 43.7% of the total noted improvement; and 33.3% of the 
department heads noted improvement. 

Can read graphs around the plant. 93% of the ABE students noted 
improvement; 41.8% of the total noted improvement; and 33.3% of the 
department heads noted improvement. This is clearly one area where 
the focus on reading graphs in the ABE curriculum had an impact on 
siudent/worker behavior. 

- lean construct a graph. 93% of the ABE students noted improvement; 
38.2% of the total noted improvement; and 1 1 .1% of the department 
heads noted improvement. Same analysis as above. 

- Can understand SPC procedures in the plant 100% Of the ABE 
students noted improvement; 49.1% of the total noted improvement; and 
44.4% of the department heads noted improvement. This is the only skill 
that 1 C0% of the ABE students say they have improved in. It is an 
important one for the same reasons described above under "need for 
Quality Control." Other students may have entered their classes with 
more knowledge of SPC than their ABE counterparts, sot heir learning 



curve is not so large. Also, as explained previously, students in classes 
other than ABE may not have focused as much on SPC in class. 

- Could take part j r Rpt. h aaEfflj tn do so. 85% of the ABE students 
noted improvement; 41.8% of the total noted improvement; and 44.4% of 
the department heads noted improvement. 

- 1 IndaffitePd my. " Rinht to KDfiUfc 85% of the ABE students noted 
improvement; 61 .8% of the total noted improvement; and 55.6% of the 
department heads noted improvement. 

Sotith Cova Manor 

The educational staff on the SCM team did an exhaustive tabulation of the data 
they had gathered and summarized the key findings as follows: 

- Mastery of skil ls critical to high-qualitv care: Students on average have 
attained 71 .35% of the key competencies aimed at by the program. 

~ Increased positive interaction hetween residents and nursing 
assistants: 

+ Residents feel they interact positively with 91 .6% of students. 
+ Residents report that 83.3% of students adequately explain 
procedures. 

+ Residents report that they now interact more positively with 

66.6% of students. 
+ Students feel that they now satisfactorily engage in 57.81% of 

the positive interactions considered critical to quality care. 

Reduced mis understandings: 

+ Residents report that they adequately understand 91.6% of the 
students. 

+ Residents report that 75% of the students adequately 

understand the residents themselves. 
+ Students report that no misunderstandings take place in 66.6% 

of their interactions with residents. 
+ 66.6 %of students report that the number of misunderstandings 

with residents has decreased. 
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Reporting changes in resident condition: 

+ Residents report that 66.6 %of students adequately relay their 
messages or report changes in the residents* condition. 

+ Supervisory staff report that 66.6 %of students adequately give 
reports to their English-speaking supervisors. 

Filling out documents: 

+ All students can fill out at least 70 % of necessary information on 
an English-only flow sheet. 

Changes in the wav SCM operates in order to make bast use of 
students' skills : 

+ Students report that 50 % of English-speaking supervisors now 

pay more attention to students' reports. 
+ The reports submitted by 66.6 % of students have enabled the 

Director of Nursing to make necessary adjustments in patient 

care plans. 

+ The reports submitted by 50 % of student s have enabled 
English-speaking licensed staff to make necessary adjustments. 

+ The reports submitted by 91 .6 % of students have enabled 
bilingual supervisors to make necessary adjustments. 

0 

Reducing dependency on interpreters and translations: 

+ 66.6 % of students feel that English-speaking staff no longer 

need an interpreter to communicate with them. 
+ The Director of Nursing has become less dependent on 

interpreting when interacting with 41.6 % of students. 
+ English-speaking nurses feel they have become less dependent 

on interpreting when interacting with 66.6 % of students. 
+ The Director of Nursing feels she does not need an interpreter 

for 50 % of students. 
+ English-speaking nurses say they donl need an interpreter 

when interacting with 66.6 % of students. 
+ Bilingual supervisors say they interpret less for 91.6 % of 

students and that they depend less on interpreting to give 

instruction for 58.3 % of students. 
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- Non-joh-raiated benefits : The general proficiency (as measured by 
the standardized BEST test) of ALL participants has developed 
considerably. 

When these findings were reported at a team meeting on July 25th, it was 
agreed that the data were "valuable," the kind of "quantitative data" which the 
team had asked for at the beginning of the study. A management 
representative noted, however, that the process of planning and carrying out the 
data-gathering and -analysis was very time-consuming and that future 
evaluations would require a less-demanding system. This same manager said 
that future evaluations should look not only at more-experienced students but 
also at participants newer to the program. 

United Electric Controls 

By late September, a total of four supervisors had completed questionnaires, 
assessing the skills of fourteen students. The consultant entered the data from 
these supervisor questionnaires into an Excel computer database system and 
produced a grid showing the average "scores" for each student, as well as the 
average "scores" for each of the eleven questions asked on the questionnaire. 
(See the United Electric Controls case study in Appendix A for a copy of this 
grid.) In the process of tabulating the data in this way, the consultant found that 
the multiple choices offered for each skill area were not as dear as they should 
be and did not readily lend themselves to numerical analysis. The consultant 
and program coordinator agreed that future such supervisor questionnaires 
should be revised to deal with this problem. 

Another design-related problem was the fact that some students were not 
normally expected to perform certain functions in their day-to-day jobs. It was 
thus not fair - nor particularly useful - for them to be assessed on these 
particular skills by their supervisors. The consultant and coordinator agreed 
that future such questionnaires should focus only on the skills most relevant for 
each student. 

The consultant and coordinator agreed that, despite these design-related 
problems, the data which were produced by the supervisor questionnaires 
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could provide rough indicators of how well students are performing particular 
Job tasks. As the coordinator put it: "While we still cant present results in terms 
of a graph or a Charles Atlas before/after picture, we should at least be able to 
digest results to show program relevance to company goals." 

The coordinator felt that "Workplace outcomes tor students at UE have not been 
promotions or big pay increases. They have been (1) success in adapting to 
constant change, and (2) contributions to company success through individual 
initiative." This view was corroborated at a July23rd team meeting where the 
first questionnaires completed by a supervisor were discussed. At this meeting, 
the company manager said that he felt that this one supervisor's responses, 
along with the students 1 responses and other evidence, indicated that the 
program has been reasonably successful in getting employees more involved. 
The team agreed, however, that the employees' English-language abilities are 
only one of many factors determining how involved workers are. Other factors 
cited at the team included the employee's "personality" and how management 
makes use of the worker's skills. 

In the consultant's view and that of the coordinator, the information which was 
gathered could be used by the instructor to further tailor instruction to each 
student's needs. The data also showed which job tasks students were not 
expected to perform at present and which tasks were commonly expected of 
students. The coordinator also noted that the process used to design and 
implement the evaluation "brought us back into contact with supervisor opinion 
and provided stronger insights into specific results (or non-results) of classroom 
projects." 

Boston City Hospital 

By August, the data from the questionnaires completed by seven students and 
six supervisors were in. The results produced a somewhat cloudy picture of 
what the impact the program had had on the students. In some cases (for 
example, communicating with supervisors, doing high-quality work, 
communicating with other staff and patients), both students and supervisors 
rated the students as doing well. In several other areas (such as taking initiative 
in staff meetings, handling stressful situations, filling out forms, anticipating 
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special needs, and thorough knowledge of their jobs) students' self- 
assessments and the ratings given by their supervisors tended to differ. It was 
also found that two students consistently were rated low by their supervisors 
even though these two students gave themselves "high marks." 

in a meeting on July 23rd, it was agreed that the fuzziness of these findings 
suggested that the instruments needed to be revised if they were to be used in 
the future. They might, for example, be edited to focus primarily on Just a few, 
highest-priority areas such as communication skills, handling conflicts, and 
dealing with emergencies. It was also agreed that future evaluations also 
incorporate data from more-traditional measures such as scores on classroom 
tests. Future evaluation work would, in any case, require a restructured team, 
with more representatives and a new program coordinator. 

Kennedy Pie Castings 

The data from the June and August supervisor interviews and student self- 
assessment questionnaires were tabulated by the consultant who used an 
Excel computerized database program. The raw data and the corresponding 
supervisor interview guide and student questionnaire are included in the 
Kennedy case study in Appendix A. 

In his two assessments of the students' Job-related abilities, the supervisor 
indicated the following: 

- On average, students fall in the "medium" range of ability. 

- They on average improved slightly during the period of June to August 
in their abilities to handle all of the given job tasks. 

- The students showed the strongest abilities in (a) identifying quality 
defects, (b) functioning as a KDC employee, and (c) comprehending 
and following directions. 

- The students were rated lowest in their abilities to (a) communicate in 
general and (b) actively participate in KDC training programs. 

- The data could be organized to provide a "ranking" of students. This 
could be used by instructors to identify which students needed more 
help and the skills areas which needed particular attention. 
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in their own self-assessments in August, the students tended to rate themselves 
on the high end. They seemed particularly confident in their abilities to 
understand their supervisors' directions, tell someone that they don't 
understand something, tell their supervisors they have a problem, and spot 
defects. 

When the above data were discussed at the third meeting of tha Kennedy team 
on October 1 , the senior company manager said that the single sheet of paper 
containing the data from the supervisor interviews was particularly useful to him. 
He felt that, when it came time to make decisions about whether to continue 
putting company money into the program, he could present this information to 
his genera) managor to demonstrate that students were in fact improving their 
job skills as a result o'» participating in the program. Similarly, the manager felt 
he could show such data to his customers who were especially concerned that 
their suppliers' (in this case, Kennedy's) workforce be able to produce quality 
goods within a given timeline. Such data were, he felt, more compelling than 
results from a standardized test. 

The manager also notea that, apart from the data produced by the supervisor 
interview, the data-gathering process had had the added benefit of getting 
supervisors and other company personnel to pay more attention to the kinds of 
goals aimed at in the education program. Tiis was corroborated by the 
supervisor who participated in the interview; he said that going through the 
interviews helped him focus on what is going on in class and how he can build 
on what the students are learning. He has seen, for example, how employees 
on his shift are more productive because they now communicate more 
effectively in a team atmosphere. He feels that the class provides a convenient 
forum for communicating important information to a number of workers at one 
time, and so he plans to continue participating as an occasional instructor in the 
class. 

Another team member - the human resources officer - similarly said that going 
over the data has made her pay more attention to the communication skills of 
the students with whom she interacts around personnel matters. The senior 
manager suggested that, given such benefits of involving supervisory-level 
employees in assessments of workers' skills, questionnaires containing similar 



questions might now be disseminated to all supervisors at the end of each 
education program cycle. He cautioned, however, that future evaluation 
activities not be overwhelming in terms of the amount of data produced and time 
needed to make sense of the data. As he put it: "Don't paralyze by analyzing." 

The senior manager felt that the supervisor's assessments of the education 
program's impacts were more meaningful than an evaluation done by an 
outside evaluator. The supervisor was seen as being "close" to the actual 
workplace situation in a way that an outsider could not be. The supervisor also 
has a personal investment in seeing that the education program in fact 
improves his workers' skills and thus approaches his assessment of the workers 
with a critical eye. (Another team member - the program instructor -- had made 
the same point before the meeting began. She said that supervisors at 
Kennedy now have more authority than they had had previously and that 
supervisors now tended to be more thoughtful and involved vis-a-v*s what 
happened on the shop floor. By going through this goal-setting and evaluation 
process, the supervisor's sense of ownership for the work process was, she felt, 
reinforced.) 

In addition to the above comments from company representatives, the instructor 
who gathered the data felt that the evaluation process was instructive in the 
following ways: 

(1) The feedback provided by the supervisor in the interviews tended to 
confirm the teacher's understanding of the students' abilities, but not 
in all cases. The instructor was able to see how a student might 
perform well in the classroom but not be transferring that ability to the 
worksite - or vice versa. This enabled the instructor to clarify where 
students' skill gaps were and to change the curriculum accordingly. 
(For example, the results suggested to her that she might divide the 
students into beginning-level and more-advanced-level groupings, 
so that beginners could get the extra help they appeared to need and 
so that the more-advanced students wouldn't be "held back.") The 
supervisor's assessments also raised questions about other factors 
unrelated to the education program which might be impacting the 
students' use of the newly-acquired skills on the job. Such variables 



might include personality conflicts with the supervisor, a work 
environment so noisy that it inhibits clear communication, and social 
pressure for an employee to "appear to understand even when he 
doesn't. 

(2) The process demonstrated to the instructor just how complex her job 
is. At a time when the entire production process is placing greater 
demands on individual workers, she is being asked to customize 
instruction to a number of job-related tasks with which she has no 
direct experience. This has led her to conclude that she needs more 
technical input from the company if she is to effectively tailor the 
program to the company's interests within the limited amount of 
instructional time available. But it is difficult to get that input at a time 
when the supervisory-level personnel who might give that input are 
being replaced and also being expected to take on other new 
responsibilities beyond those required by the education program. 
(This point about the complex nature of what she has to teach was 
made particularly clear when she tried - unsuccessfully - to set up a 
simulation activity as one way of assessing students' abilities to 
handle particular job tasks. She had thought that the problem to be 
solved in the simulation was relatively straightforward, but soon 
learned that even seemingly-simple tasks could be solved in a 
number of ways depending on details which she could not foresee 
when designing the simulation problem. ) 

(3) The goal-setting process used in the initial steps of planning this 
evaluation was very useful for the instructor. The extra attention paid 
to setting clear goals this time has shown her that, by setting clear 
goals, she can more effectively monitor progress toward those goals. 

Step 9: Reporting of data and other follow-up action (August - November 
1991) 

Now that they had summarized and analyzed the data, the six teams were faced 
with the task of preparing reports for the audiences they wished to present their 
findings to. In consultation with the consultants, the teams decided to whom 



they would report their findings and the formats of those reports. These 
audiences, the form the reports were to take, and other foitow-up action varied 
from program to program, as follows: 

Hampden Paner 

The coordinator presented a written summary report to the members of the 
Hampden Paper evaluation team prior to the fourth and final meeting of the 
team. That report generated discussion in the fourth team meeting, and was the 
"skeleton" of the final report which the program coordinator then completed in 
November. The final report will be distributed to supervisors and other 
managers at Hampden Paper in order to share information about the education 
program and to generate support for it. 

Aerovox, 

In the third site visit meeting in mid-September, the team agreed that it should 
present a summary report to three or four Aerovox vice presidents, especially 
the vice president for quality control. The vice president for human resources, a 
member of the evaluation team, suggested that selected findings could also be 
included in the Aerovox Annual Report as well as in the Aerovox newsletter. He 
took responsibility for making this happen. The coordinator invited the other 
VPs to the fourth and final team meeting in late October, and presented them 
with a summary report. 

When the team members reported their preliminary findings to the vice 
president for quality control at the fourth site visit meeting, they found a 
surprisingly sympathetic ear. Recently returned from a visit to another company 
which had embraced the "high productivity" model (team-based work, building- 
to-order, total commitment to quality, and worker involvement in the work 
process, etc), he was especially interested in worke r s' reports that participation 
in the Aerovox education program helped them to develop a sense of team 
work and team identity. (This is included in the ASTD categories: 71 .9% of the 
total reported improvement; 55.5% of the department heads reported 
improvement.) 
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He suggested that future education programs might focus more on skills like 
team-building and on changing the organization of the company in general 
(with significantly more emphasis on valuing the ideas of workers), as well as 
on more-traditional basic skills. The evaluation team meeting provided the 
venue for the vice president for quality control to discuss this new idea with 
other key people in the company and to begin to plan how these new ideas 
might be carried out. 

The final report of the Aerovox evaluation, which the coordinator completed in 
late November, wiH be distributed to the other vice presidents and to other 
interested parties as a way of promoting support for workplace education. 

South Cove Manor 

In a meeting with the consultant on July 25, the team agreed that the above 
findings would be presented - in a combination of oral form and written 
summaries - to the following audiences: the SCM board, the board of the 
Chinese American Civic Association (the education service provider), students 
in the SCM ESL class, and the state office which provided funds to the program. 
(Note: Subsequent to the July 25th meeting, SCM decided to terminate its 
contract with CACA and close the ESL program. It was thus not clear at the time 
when this report was written whether the program would make further use of the 
data gathered, the instruments used, or the team structure which had been 
developed. However, CACA was actively planning to use a similar planning 
and evaluation process in the workplace education programs it was setting up 
with other area employers.) 

Kennedy Die Castings 

At its third meeting on October 1, team members agreed that they would 
summarize the data and conclusions described under Step 8 above in the 
following ways: The senior manager would combine the above findings with 
additional comments of his own and from his human resources officer. This 
report would then be available when management decides what amount to 
invest in the education program in the future. The education providers likewise 



will Incorporate the study's findings Into the final reports they submit internally to 
their own supervisors and externally to their funders. 

These same educators (who are based at Quinsigamond Community College) 
are also moving to transfer the experience gained in this year's outcome study 
to their work in other workplace education programs In the Worcester area. For 
example, because the program for workers at the T.J. Maxx warehouse is now 
becoming increasingly work-related, the Quinsigamond educators hope to 
implement a goal-setting process which will help TJ. Maxx to clarify just which 
job-related goals it wants the curriculum to focus on. Similarly, the Norton 
Company is using its new education program as a way of instituting a more 
participatory work format. The company is thus interested in using the "team" 
goal-setting and planning approach which the Quinsigamond educators are 
developing. The Quinsigamond staff will use information about the process 
used in the Kennedy project in their communications with these and other 
employers. 

United Electrical Controls 

When the data from the four UEC supervisors and nineteen students were 
analyzed as described under Step 8 above, the coordinator agreed that, with 
design revisions, the questionnaires might be used again for additional 
evaluations at UE. The coordinator said that he would like to combine these 
types of questionnaires with other instruments he had previously developed or 
considered at UE and other programs. (These other instrumb.tts could include 
multiple-choice job-related tests for students and roundtable discussions in 
which supervisors assessed what was happening in the program.) When 
organized in a "battery" or "portfolio," these data-gathering instruments would 
provide a program coordinator with a set of evaluation tools to adapt for use in 
any workplace program. In the meantime, t e coordinator agreed to provide a 
copy of the "grid" of supervisor responses to the team's human resource 
manager. This grid could serve both as an indication of the rough findings of 
the supervise: questi onnaire and as a model for how similar data could be 
organized in the future. 
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Boston City Hospital 



At their July 23rd meeting, representatives of the BCH team agreed that they 
would summarize what data were available from the two questionnaires and 
student test scores and present them to senior management This would be . 
done to (1) increase management's support for the program in general and (2) 
get senior management more involved in the team. The latter involvement by 
senior managers was seen as important because management needs to have 
input into program goal-setting, coordination of the basic skills program with 
other human resource programs, and other key activities. 

The team suffered further attrition in September, to the point where only one 
representative from the original team was actively attending to educational 
issues. This "sole survivor" (the union representative) said in a letter to the 
consultant that , despite the damage done to the team by staff reductions, she 
"would continue to push for more commitment to workplace education.. .and 
suggest that we use some of the tools that you have left us with in terms of goals 
setting and evaluation." 

Step 10: Evaluation done of the outcome study process itself : September - 
November 1991 

In order to clarify how effective this outcome study process was, the consultants 
provided opportunities for the six evaluation teams to assess their own 
experience in the project and to make recommendations for possible future 
outcome study activities. The consultants informally asked various evaluation 
team members how effective they thought the evaluation process had been and 
where there was room for improvement, in addition, the consultants constructed 
a brief questionnaire which asked the team members to assess the 
effectiveness of the evaluation process and make recommendations for 
improvement. A copy of this questionnaire is included in Appendix B, under 
separate cover. 

The consultants' conversations with team members and the written evaluations 
indicated that, in general, the process of establishing an evaluation team which 
represents the partnership and together defines appropriate outcomes is very 
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valuable. These evaluations also indicated that more work needs to be done in 
defining outcomes, designing the best measurements for these outcomes, and 
in balancing the need for effective evaluation with the demands it places on 
everyone's time. This feedback has been incorporated into the analysis 
inscribed in Section III below. 

III. Looking Back and Looking Ahead: An Assessment of the Project and Its 
Implications for the Future 

Because this project was framed by its planners as experimental in nature, it is 
important now at its closure to assess what in fact was achieved in the project. 
From this self-examination, we can get a clearer picture of how the workplace 
education field might build on the experience provided by the project. 
This section first assesses how well the project met its stated goats. It 
concludes with a set of recommendations for the various parties which might 
have an interest in building on the project's experience in the future. This 
discussion incorporates feedback received from the evaluation teams along 
with the consultants' observations. 

What the Project Achieved: 

As stated at the beginning of this report, the MWEI Cycle 4 evaluation was 
originally set forth as "a combined outcome study and organizational 
development project," with the following three major goals: 

1 . Build the capacities of individual programs to conduct outcome 
evaluations. 

2. More clearly define desired outcomes for each site and produce data 
which demonstrate what in fact is being achieved. 

3. Develop a "tool kit" (an evaluation planning process, data-gathering 
instruments, and resource persons) which other programs can use to 
develop their own planning and evaluation systems. 
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What follows are assessments by evaluation team members and the 
consultants of how fully each of these goals was achieved: 

Goal #1 : Build the capacities of individual programs to conduct outcome 
evaluations. 

Achieving this goal was key to the success of the entire project. In general, the 
goal was achieved, although the level of achievement varied from site to site, 
and even within individual sites. 

The six teams carried out the planning, instrument-design, data-gathering and 
analysis, and reporting activities described in Section II above. In the process, 
most of them emerged with a clearer team identity and an understanding of how 
they might not only assess what is being achieved in their programs but - from 
the start -build their education programs around clear goals reflecting the 
diverse interests represented in each program. Team meetings gave 
supervisors, managers, workers, union representatives, and education staff the 
opportunity to meet regularly, focus on a common purpose, and draw on the 
strengths of their different roles. Often the evaluation team meeting was the 
only time that representatives from these different groups gathered together to 
discuss the education program. This means that issues which needed to be 
discussed but had no other place in which to be discussed were reviewed and 
resolved. 

• For example, at Hampden Paper the union representative had the 
opportunity to query the company president about the effects of an 
educational certification program on seniority, and a supervisor reported 
candidly to the president about the critical attitudes of other supervisors 
toward the education program. These issues were highly relevant to an 
effective evaluation but also to effective program development and 
implementation. Without the evaluation team meetings, this important 
information would not have been shared in a timely way -- and maybe 
not shared at all. 

• Similarly, at Kennedy, the team meeting triggered goal-setting meetings 
with other supervisory staff. Rather than just rely on outcomes defined in 
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the evaluation team, the coordinator decided to query other interested 
parties, too, and to involve them in a goal-setting/feedback process. The 
purposes of these goal-setting activities were to develop the curriculum 
as well as to enhance the evaluation. 

• The process of defining outcomes for the education program helped a 
manager at United Electric Controls to see to see how the standards set 
for workers in the ESL program also apply to other workers. The team 
process thus informed the larger human resource development system 
which the company was introducing and helped it to flesh out standards 
which all workers should be measured against. 

For the last three years, the MWEI has required that each funded program be 
overseen by on advisory board which represents the diverse groups in the 
program partnership. These groups typically consist of management, labor, and 
the education provider. While these advisory boards were originally intended to 
provide opportunities for open exchange, they have often become forums only 
for upper-level managers to be kept informed about program details. Real 
discussion in most of the existing advisory boards is limited, especially by the 
roles that govern relationships in the workplace. In the new evaluation teams 
created for this year's outcome study, however, the difference in roles and 
perspectives was valued and actively elicited by the consultant. In contrast, the 
meetings of the older advisory boards have not been facilitated by an outside 
evaluator whose aim is to include all perspectives in the discussion. 

Other positive results of the team-building effort were (1) the development 
among some team members of new skills and self-confidence vis-a-vis program 
evaluation and (2) clarification for students about how evaluations can be used 
and resulting increased willingness among students to participate in similar 
evaluations in the future. 

The teams' abilities to make maximum use of their new identities and 
knowledge, however, were limited by obstacles which were both built into the 
evaluation plan and which were the result of organizational problems or 
changes in the companies. These obstacles were as follows: 



1. The evaluation plan required the consultants to simultaneously 
develop the team and use the team to perform the required work. None 
of the teams had worked together as such before the January 10th 
meeting. While there was good will and desire to work, teams do not 
become teams because someone tells them they are a team. Rather, the 
team builds an identity over time, through planning their work and then 
working together. The time constraints of the evaluation, and the limited 
number of site visits which the consultants could make to each program, 
overly compressed the work to be done. 

2. Team members who did not understand the importance and value of 
working as a team could not take ownership of the evaluation in a way 
that insured their ma v imum participation. 

3. Some team members had a traditional mindset vis-a-vis "evaluation," 
seeing it as something done by outside experts rather than by program 
participants themselves. 

4. Some team members had limited experience of this type of 
collaborative evaluation, felt they did not have the skills to do the work, 
and were sometimes confused by what they were asked to do. 

5. Team members often found it difficult to come up with mutually- 
convenient times when they could work together. 

6. Some programs were operating within organizations experiencing 
considerable pressure due to budget cuts and staff cutbacks, leading 
team members to question the value of the evaluation. 

7. The two consultants had only limited lime to help the programs deal 
with these limiting factors as they emerged - often unpredictably ~ 
during the course of the project. 

Goal #2: More clearly define desired outcomes for each site and produce data 
which demonstrate what in fact is being achieved 



The teams made real strides in nine months toward, first, Identifying meaningful 
goals for the program and, then, designing and using appropriate instruments to 
measure progress toward those goals. However, the data which resulted from 
these efforts were of varied quality and usefulness. 

Most members of the teams went into this project believing that, yes, good 
things were happening in their education programs and that they simply 
needed to find an appropriate, meaningful way of documenting these positive 
results. This proved to be much easier said than done, for two reasons. In part, 
putting together a sound study was hindered by the "obstacles" cited under the 
discussion of "Goal #t" above. But this study of outcomes was also difficult 
because - even under the best of circumstances - It is difficult to make tangible 
the impact of an education program on a particular worker's behavior on the 
job. These difficulties stem from the fact that employee behavior is determined 
by so many factors be> and the education program, standards for worker 
performance are not clear, and other variables. (See Sticht in References.) 

The best-quality data gave the clearest, best-balanced picture of how fully the 
learners were currently performing certain agreed-upon functions. The teams 
recognized early on that they would have to rely primarily on the assessments 
of people involved with workers on the job ~ supervisors and "customers" - and 
on workers' own self reports. The reliance on these assessments stemmed 
from the relative ease with which they could be designed and completed. 
These assessments had a built-in degree of subjectivity to them. The teams 
attempted to deal constructively with this subjectivity by following principles of 
social science research, particularly that of triangulation: (1 ) setting clear terms 
and standards for the abilities being measured and (2) relying on a number of 
informants' perspectives to inform the evaluation rather than on, for example, 
just one supervisor's assessment or just the students' self-assessments. The 
"simulations" developed by the South Cove Manor evaluation team were an 
attempt to identify more objective measures of student progress. While 
simulations might not be appropriate in all work settings, most teams realize the 
importance of establishing more-objective measures of student progress which 
complement the worker self- reports and reports by supervisors and customers. 
What "more-objective measures" means in each program needs to be defined 



carefully by each evaluation team, without discounting the value of soliciting 
perceptions of change from the players involved. 

Despite these honest attempts to grapple with issues of subjectivity, the quality 
of the data which came back were often of limited scope. For instance, one 
team could, due to staffing changes, come up with only one supervisor who 
could give informed feedback on (earners' behavior and that single supervisor 
had interacted with the learners only recently rather than over the full fife of the 
program. Also, much of the data produced gave only an indication of where 
particular students stood at present, and there was thus only a limited basis for 
comparing their current abilities to their skill levels when they entered the 
program, apart from perceptions of change reported after the students had 
already gone through the program. (Kennedy Die Castings was the only 
program which collected something like longitudinal data by having a 
supervisor assess workers' abilities twice over a three-month period.) 

In such cases, the data which came back had to be viewed as a base line 
against which future similar data could be compared. This initial stab at 
gathering data is viewed positively as a pilot effort from which the teams could 
get a clearer picture of which outcomes to focus on and the most effective ways 
of gathering information pertinent to those outcomes, and then revise their 
existing instruments and perhaps develop new ones. 

In the best cases, teams focused on a carefully-selected set of outcomes and 
then were able to gather a sufficient amount of data about a significant portion 
of the student body from a number of sources. The resulting data, in turn, 
provided a sufficient body of reliable evidence to draw conclusions from. 
Coming up with such extensive data required much hard work by committed, 
thoughtful team members who went the extra mile in tracking down sources, 
testing and revising instruments, and developing a systematic way of recording 
and analyzing results. 

In both the stronger and less-strong efforts, the data which were gathered had - 
at a minimum - the potential for getting team members to be clearer about the 
goals their programs should be aiming at, even if the teams had not yet 
developed smooth mechanisms for assessing progress toward those goals. 



Goal #3 : Develop a tool kit" (an evaluation process, data-gathering 
instruments, and resource persons ) which other programs can use to develop 
their own planning and evaluation systems. 

As stated in the discussion of Goal #2 above, the stronger teams were fairly 
successful in developing a process, instruments, and a pool of resource 
persons which they could use to carry out similar outcome studies in the future 
and to teach others how to conduct similar studies . (There has emerged a 
multiplier effect beyond the scope of this project For example, The Chinese 
American Civic Association is now interested in using the SPSS program to 
computerize evaluation data on all the other education services it provides in 
the Chinese community. Quinsigamond Community College is now adapting 
key elements of this year's project ~ goal-setting and other planning and 
evaluation activities - to its work with other area employers.) These processes 
and instruments were not perfect, however, and they would require additional 
supports to take the next step of strengthening the fledgling evaluation systems 
they had begun to develop. These supports would include "internal'' ones like 
paid time for team members to devote to evaluation work as well as active 
participation and commitment from all other team members. Support would 
also have to come in the form of "external" assistance such as workshops which 
focused on the specific technical needs of team members. (For example, many 
teams could likely benefit from guidance on new approaches to data-gathering 
which go beyond the traditional questionnaire and checklist.) 

How the positive experience of these teams could be built on within the six pilot 
programs and by other workplace education efforts is discussed in the following 
Section. 

Recommendations for Future Efforts Aimed at Strengthening the Planning 
and Evaluatio n Capacities of Workplace Education Programs 

The following recommendations are divided into two sets. The first set is for 
state-level planners and funders; the second is for the evaluation resource 
people who would provide technical assistance to workplace planning and 
evaluation teams. 
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Recommendations for state-level planners and functors: As Stated previously, 
this year's outcome evaluation project has demonstrated that efforts targeted at 
the project's three goals can produce significant benefits for workplace 
education programs. Benefits include strengthened planning and evaluation 
capacities for individual workplace education programs as well as clearer data 
about what is in fact being accomplished in the programs. This year's project 
showed that, to be successful, future efforts should build on this year's 
experience. A future statewide effort would focus on putting the key elements of 
successful on-site program evaluation in place while dealing constructively with 
the obstacles which block successful evaluation team efforts. 

In order to build on the strengths of this year's experience and avoid the 
problems encountered, state-level planners and funders of workplace 
education programs should: 

« Modify funding guidelines to require that each site have an active 
planning and evaluation team in Place if it is to qualify f or funding. 
These teams would be more than figurehead groups. They would be 
required to work together over the course of the funding period to 
perform appropriate planning and evaluation activities. They would 
need an active coordinator in charge of organizing planning and 
evaluation activities. These activities would include ongoing goal- 
setting activities and development and use of appropriate measures of 
program outcomes. Program coordinators are the likely people to take 
on this role. They have been responsible for evaluation activities in the 
past and actively participated in the teams this year, taking the lead in 
organizing teams and collecting data, in subsequent years, this work 
should be integrated into the responsibilities the coordinators are 
expected to carry, rather than be seen as extra work. 

* Require these planning an d evaluation activities to follow a step-bv-step 
process similar to that developed in this first year's project. With proper 
training of key team members, as well as access to a core group of 
resource persons with technical expertise in the various facets of 
planning and evaluation work, the teams could efficiently incorporate 
planning and evaluation as a central function of their work. Already- 
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existing teams would continue to receive technical assistance from 
resource persons. 

• Organize and support a cadre of resource persons who would provide 
technical assistance to workplace teams. The state-level partners 
would thus not only write such a new emphasis on planning and 
evaluation activities into funding guidelines, but would also organize 
the core group of resource persons referred to above. These resource 
persons would hold formal training sessions in evaluation for 
workplace educators from around the state and also be available for 
on-site consultation and informal telephone feedback sessions. Print 
and other resources - like this report, copies of sample instruments 
developed this year, and by others, relevant reports and videos from 
around the nation, and annotated bibliographies- might also be made 
available.directly through the resource people, the MWEI Coordinator, 
and the System for Adult Education Support (SABES). 

* Provide additional workshops where supervisors, managers, workers. 

union representatives, and education staff meet together in their 
interest groups across programs. The outcome evaluation this year 
only touched on the benefit of this kind of cross-fertilization in the 
January 10th workshop. With the exception of education staff who 
already enjoy regular meetings with their counterparts from other 
programs, all of the participants at that workshop said that they wanted 
more time with their interest groups. 

Recommendations for evaluation resource persons: in addition to what the 
state-level partners need to do, a good deal of activity is needed at the level of 
the local workplace education partnerships. If the partnerships are re- 
organized under "planning and evaluation teams," then the members of those 
teams must be educated about the value and technical requirements of effective 
program evaluation. Team membership should be limited to those who have 
the desire and time to work. Teams should be discouraged from "dumping" 
planning and evaluation activities onto one or two team members' laps. The 
resource persons should help team members to see the required activities as 
the responsibility of all team members. Evaluation resource persons should 
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encourage team members to see that each of them represents an important set 
of interests in the program, and to use well-supported planning and evaluation 
activities to insure that the program serves their respective interests. 

For the planning and evaluation teams to work, all team members need to take 
on ownership for the process. Then they need to commit the time and effort 
necessary to figure out what an effective planning and evaluation process 
requires, and see that it gets done. 

Specifically, to best facilitate team-based evaluations in the second year, 
evaluation resource persons should: 

• Provide training/organizational-development opportunities in which 
teams can develop a team identity and negotiate goals to be achieved 
through collaborative program evaluation. 

• Develop exercises in which teams can clearly define desired workplace 
education outcomes. 

• Actively address the contradiction between programs' desires to 
measure meaningful outcomes and the limited amount of time they feel 
they can invest in collective planning, data gathering, and analysis. 

• Explore different ways to collect data beyond the checklist, survey, and 
interview; develop a "menu" of methods for use across programs, 
including both "direct" and "unobtrusive" methods. 

• Consult with a specialist in measurement to ensure that evaluation tools 
are methodologically correct. 

• Gather all the teams together for more frequent sharing of information. 

• Expand the computer data-analysis expertise among teams for the 
following reasons. Computer analysis can: 

- help to systematize the sequence of steps in conducting an 
outcome study; 
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establish a system at the beginning of a study - for even a small 
group of respondents - that can be expanded and compared over 
time; 

- over time help to bring other data-collection procedures in the 
workplace into a comparable format; 

- present data in a statistical format which appeals to managers and 
others who want "hard" information about how the program is 
working; 

- provide another way for students to become involved in their 
programs and their evaluations. 

A second year of planning and evaluation activity could thus combine 
guidelines, funding, and technical assistance to create a new orientation and 
new capacities in all workplace education programs in the state. This second 
year's activity should itself be evaluated and in turn built on in future years. 
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Curriculum Study 



I. Introduction: Why the Curriculum Study was Requested 

In the fall of 1990, as part of the fourth year evaluation, the Massachusetts 
Workplace Education Initiative state level partners requested a study of the 
curricula at a sample of workplace education programs. Through this study they 
hoped to: 

• Better understand curriculum, and the process and state of the art of 
curriculum development, as they exist in the Initiative (see pages 83-91); 

• See what curriculum features appeared to be effective, and what curriculum 
practices were most promising (pages 122-127); 

• See the range of similarities and differences in curriculum across the 
programs (these are addressed throughout this report); 

• Better understand what skills, knowledge, and experience workplace 
education teachers need, and what their jobs require (pages 118-122); and 

• Receive recommendations on how curriculum might be improved at individual 
sites and across the initiative (pages 131-137). 

They also wanted to know: 

• What philosophies and educational approaches were being used (pages 83- 
91); 

• Whether or not programs had curriculum objectives, and if so, if the objectives 
were being measured (addressed in outcome study); 

• Whether the curricula addressed workers' needs or only employer needs 
(throughout this report, it is clear both are addressed); 
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• Whether or not learner-centered curricula were being used (throughout the 
report there Is evidence that to varying degrees the curricula are learner- 
centered); and 

• If workers were able to participate more fully in the workplace because of the 
curricula (throughout the report there is evidence that they are, although to 
varying degrees). 

Finally, they wondered how these programs would compare to an emerging set 
of norms articulated in such documents as the Massachusetts Department of 
Education's most recent request for proposals, the Massachusetts Interagency 
Literacy Group's "Principles for Effective Literacy and Basic Skills Programs," 
and the Workplace Education Initiative Guidebook and Request for Proposals. 

II. Curriculum Evaluation Methodology: How the Study was Carried Out 

In the Fall of 1990, we developed a set of norms from the documents which had 
been identified by the state level partners. (See Appendix C, under separate 
cover, for "Workplace Education Initiative Curriculum Standards.") 

From these norms, we developed data collection instruments: a general 
information questionnaire, administered prior to the site visits; a format for 
review of curriculum materials; and a set of group and individual interview 
protocols for gathering information from coordinators, teachers, evaluation team 
members and students. (See Appendix C for copies of the protocols and other 
data gathering instruments) 

We visited four programs, each for at least two days, in the winter and spring of 
1991 in order to observe classes; interview students, teachers, coordinators, 
evaluation team members and others; and to review additional curriculum 
materials. 

After completing all the site visits we analyzed the information from these 
interviews and summarized our findings in this report. 
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The workplace education programs studied here, like fast rivers, are dynamic, 
changing entities. This curriculum study was an opportunity to jump into these 
rivers and float with them each for a brief period. By the time we were writing the 
draft of this report, just months after our data collection, we were already aware 
of significant changes being made in the programs and their curricula. This 
report should be viewed as a collection of snapshots of the Workplace 
Education Initiative program curricula at a time when the programs were young, 
and growing fast. 

IR. Findings 

A. The Meaning of "Curriculum" at the State Level and at the Individual 
Program Level 

"Curriculum" in this study, as defined by the state partners, means The 
materials developed and used by program staff, the process by which materials 
are put together, actual classroom practices, and the philosophy which guides 
these activities." This was a definition agreed upon by the partners at the 
study's outset. Assessment is not included in this definition and is only touched 
upon in this study. It is an important area for program evaluation, but was both 
beyond the scope and resources available this year. 

This definition represents the anticipated diversity of the curriculum definitions 
and approaches used by the programs in the Initiative. One of our purposes 
was to better understand what each of the programs meant by "curriculum" and 
how diverse, in fact, these definitions were. 

Curriculum and Curriculum Development at Kennedy Die Castings. Inc. 

The philosophy or approach which undergirds ESL curriculum development at 
the Kennedy Die Castings, Inc. program, is described in the words of its 
Coordinator, Kathy Rentsch, and Head Teacher/Curriculum Developer, Kathy 
Soderstrom, as: "a multi-faceted process that develops all language areas 
simultaneously using the strengths students have, and building upon them in a 
natural, holistic manner. It is based on the belief that communication is 
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composed of the elements of speaking, listening, reading, and writing, and that 
they should be developed naturally and together." 

"Students are encouraged to use language that is familiar to them. Lessons are 
student-centered and always taught.in a natural language context, not in 
isolated drills. The processes are taught as a whole, not as sub-skills. The 
workplace and Its specific situations - machinery, processes, personnel, rules 
and regulations provide the necessary background knowledge." (From 
"Introduction to Kennedy Die Castings, Inc. Curriculum") The ultimate goals of 
such an approach - for student/workers - are increasing their self-confidence; 
helping them to function effectively in the workplace and outside; and "enabling 
and empowering" learners through improved English language skills to 
articulate their views, read, write, compute, participate and work in groups, and 
solve problems, especially job-related problems. 

In this program, "curriculum" refers to the units and materials developed or 
adapted, and the work tasks and daily living tasks from which the language 
lessons are drawn. It is also the instructional process which tailors the learning 
to the variety of backgrounds, experiences, skills, knowledge, and learning 
styles of student/workers. Thus, the line that is sometimes drawn in education 
between curriculum and instruction is here deliberately blurred to avoid 
fragmentation and to assure that the curriculum is learner-centered. 

There are 29 work-based curriculum units, in various stages of refinement, 
derived from student language, work, and life skill needs as articulated by 
company representatives and by workers. They include topic areas such as: 
health and safety; quality assurance; the process, tools, and equipment of 
aluminum die casting; reading signs at work; and communication skills. The 
curriculum units are intended to guide, aid, and assist teachers as they use and 
adapt them to facilitate learner-centered, contextual activities in the classroom. 
The curriculum development process used by the Worcester Workplace 
Education Collaborative, which includes Kennedy Die Castings, Inc. as one of 
its sites, is as follows: 
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1 . The curriculum developer observes what learners do at work, takes 
photographs, and collects "realia" such as notices, signs, and written 
instructions which are part of work tasks or the work environment; 

2. She facilitates a collaborative goal-setting session with learners and with 
supervisors to determine overall interests and goals; 

3. From these, she develops themes which address worker language skills, 
needs and interests such as: signs in the workplace; health in the workplace; 
quality assurance; and work process and product control; 

* 

4. These are reviewed by teachers, and by the site teams, which include 
representation from the company, including supervisors. Curriculum is not 
reviewed formally by students. 

The curriculum units are tailored by the instructor in the classroom using such 
co-operative learning techniques as peer grouping, role playing, discussion, 
experience charting, picture stories (using site-specific photographs), problem- 
solving, and student-written booklets such as "A Day on the Job of...." or "The 
machines and Processes at...." 

The curriculum is organized by theme (work-related or other contextually- 
related theme such as: health and safety; quality assurance; communications; 
aluminum die casting; casting defects; tools, equipment and gauges; signs at 
the workplace, etc.) instead of by a sequence of language skills or life skills. 
Language skills, however, are included; they are embedded in the themes. 
"Theme," as we understand its use in this program, means a contextual topic, or 
an area of student concerns, problems or issues. 

Classroom activities, however, are not limited to the content of these curriculum 
units. Instructors also draw upon students' immediate language needs from 
work and from their lives outside work which students, the instructor, or 
supervisors may identify. 

Although curriculum revision is an ongoing process, in the summer of 1 990 
there was a major revision of the curriculum. The former curriculum separated 
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"work-related" learning from "ESL" learning. The new curriculum integrates 
these, and Is used at all three companies in the Worcester Workplace Education 
Collaborative. It consists of curriculum units and a teacher's guide, and is 
photocopied and given to students to take home. 

Curriculum and Curriculum Development at Bos tnn Citv Hospital 

Boston City Hospital has two adult basic education courses with quite different 
aims and curricula: the basic academic skills (ABE) course and the pre- 
technlcal science and study skills course. The curriculum for the basic 
academic skills course is a set of competency areas and specific competencies 
or "performance indicators". It has two sections: "Healthcare Workplace Skills," 
and "Academic Skills." The first section is largely workplace-specific basic 
skills, the second a continuum of basic skills leading to the goal of passing the 
high school equivalency diploma (GED) exams. 

The first section of the basic academic skills course, "Healthcare Workplace 
Skills," includes such competency areas as: telephone communications, e.g. 
taking and giving messages in hospital units/floors and in outpatient clinics, or 
responding to an emergency; forms, e.g. accurately completing applications, 
surveys, health information forms, and incident reports, and filing a grievance 
with one's union; Job Management Skills, e.g. understanding the purpose of 
one's department, balancing resources, cultural sensitivity issues, and others. 

The second section, "Academic Skills," includes: mathematics, reading, writing, 
science and test-taking skills. Basic math skills range from addition and 
subtraction of whole numbers through algebra, and include some life skills such 
as map reading and household budgeting. Reading skills range from word 
recognition through inferential comprehension, critical reading skills and 
interpretation of literature. Writing skills include work-related writing, e.g. 
charting medical records, as well as GED test essay writing skills. Science 
topics are both GED preparatory and health-related. This class and its 
curriculum are intended to cover a wide range of skill levels from functional 
illiteracy through secondary school level skills. 
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The pre-technical science and study skills course and curriculum is intended for 
workers going on (or back) for further academic and career training, it is a "next 
step" course for those who have secondary level skills but do not have the study 
skills and/or the science knowledge to enroll in or do well in post-secondary 
training courses. 

Study skills such as: how to listen to a lecture, or how to use the 'PQRST 
method of study, and test-taking skills are taught and practiced. The content of 
lectures, textbooks and handouts is from the health sciences: basic anatomy; 
basic physiological chemistry; chemistry, matter and life; basic organic 
chemistry and cellular biology; tissues, glands and membranes; disease and 
disease-producing organisms; and the various body systems. A last section 
prepares students for writing a college-level research paper and includes the 
mechanics of organizing research data and writing a research paper through at 
least two drafts. Because this is a college preparatory course students must 
also pass a written final examination as one of their competencies. 

Despite what appears to be a rather set syllabus for each course, when asked 
what the education program's philosophy or instructional approach was, the 
program's Teacher/Counselor/Coordinator, Paul Guldenzoph, said that he tries 
to identify employee learning needs from the perspectives of students and 
supervisors and to "make a flexible attempt to meet their needs." Where 
perceptions of these needs differ, he said, he tries to strike a balance. 

Curriculum and Currinulum Development at Hampde n Paner Company 

At Hampden Paper Company "curriculum" includes a syllabus of competencies, 
the sequencing of that syllabus, learning activities, materials and, and other 
parts of the content in the classroom. The syllabus consists of two sets of 
competencies: a "skills checklist", a relatively new basic set of learner 
competencies which the teacher/coordinator derived from assessing student 
needs from the point of view of both students and supervisors; and a 
competency-based GED preparation curriculum which was developed by Jobs 
For Youth-Boston, Inc. 



The Fail, 1990 version of the checklist includes reading, writing, math and three 
job-specific competencies. The six math skill areas include: the four basic 
operations, simple percentages, simple fractions, reading a ruler, and using 
liquid measures. Reading includes: a formula ticket, graphs, charts, political 
maps, road maps, classified ads, and understanding main idea of a selection at 
a certain grade level. Writ'ng skills include: filling out forms, completing formula 
tickets, using the word processor, and writing a paragraph and an essay. Job- 
specific competencies include: giving and following oral and written directions, 
and explaining one's machine and one's job. 

The GED competencies include those required on the five GED tests: literature 
and the arts; mathematics, science, social studies, and writing skills. 

Another way in which the program looks at "curriculum", and one emphasized 
by Teacher/Coordinator, Janice Rogers, is that because each individual student 
has his or her own curriculum, created for that person from individual student 
conferences, and also incorporating competencies tailored from the program 
syllabus, it also makes sense to refer to the program as having many curricula, 
one for each student. 

Unlike the other three programs in the curriculum study, the Hampden Paper 
Company workplace education program is open-entry, open-exit, and 
individually-paced. Student/workers arrange appointments for study time in the 
learning center (classroom). Their learning is supervised by a teacher who is 
available to provide each student with individual instruction. Because there is 
great interest in meeting individual student needs, especially those which arise 
from work tasks, we saw many instances where a curriculum had been 
developed for each individual student from that person's articulated needs in a 
conference with the teacher. The "Chemical Mixer Formula Ticket," which will be 
described later, is an excellent example of this. 

In this program, especially, the notion of "curriculum," as well as the content is 
evolving. The introduction of computers in the classroom this year, for example, 
appears to have generated some new curriculum dimensions: for example, 
word processing, simple database filing programs, and the use of 
spreadsheets. As planning in the company moves forward, and employees are 
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encouraged to prepare for and demonstrate new work area competencies, no 
doubt the focus of the curriculum will grow and/or change again. 

Curriculum and Curriculum Development at South Cove Manor Nursing Home 

Initially the education staff at South Cove Manor developed a competency- 
based curriculum. Halfway through the first year of the program, in mid-1 989, 
they radically modified their approach to curriculum development. Johan Uvin, 
the On-site Coordinator/Instructor, has written extensively about why this 
change was needed, and about the process for curriculum development they 
now use (Caidwell/Chang/Uvin, 1991). Some of the information in this section 
is based on his writing, some on our interviews. 

To understand "curriculum" at South Cove Manor, requires some understanding 
of the original curriculum, the needs which led to the adoption of a new 
meaning and process for curriculum development, and that process itself. At 
the cost of oversimplifying, we will briefly attempt to do this here. 

At the Chinese American Civic Association, the education partner for this 
program, the tradition of curriculum development is competency-based models. 
When this program was first funded through the state and federal Workplace 
Education Initiatives, it was assumed that a teacher-developed, competency- 
based curriculum would best meet the needs of students in this situation. It was 
organized around language and work task needs, as determined by teachers, 
and was also organized by "Student Performance Levels", "Literacy" levels and 
"arithmetic Computation" within each curriculum unit. 

For example, in a section of the original curriculum called "Things Nursing 
Assistants Need To Know How To Do For Their Job" is the competency area 
"Provide For Safety, Privacy, Comfort, and Well-being." This includes six 
competencies, such as "answer call lights and call bells", "respond to 
behavioral problems," and "relate to visitors and family." And each of these is 
broken down into very specific competency statements (usually English 
language competency) such as: SPL 1 "to ask for clarification by asking simple 
yes/no questions" or SPL 4: "to identify and explain patient rights, abuse, 
mistreatment and neglect using the patient bill of rights as a reference." 
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As the curriculum was being written, Johan Uvin and his teaching colleagues, 
Marianne Caldwell and Hsiao Chang found many shortcomings in this 
approach: 

1 . Soon after they were written, whole sections became obsolete, because in 
this work environment job descriptions change very rapidly. Like many 
dynamic work environments, changes are driven by up or down-tuming 
economics and labor force supply and demand, new policies, new forms, 
and diminishing or expanding sources of education program funding. The 
competency-based curriculum, they felt, could not quickly enough absorb 
and respond to these changes; 

2. The logic of the competency-based curriculum, organized in performance 
levels, did not correspond to the reality of the classes, which were multi- 
level; 

3. The stuaent/workers had a very wide range of healthcare skills; some had 
acquired and practiced medicine in China. And they had a wide range of 
differing interests, needs and goals. Furthermore, with each new class of 
students, the needs and interests of the groups changed dramatically. A 
curriculum which in theory attempted to predict the needs and goals of 
learners in advance of a given class proved not to be useful in reality; 

4. Throughout the competency-based curriculum, the reference point was the 
skills to be learned. White both teachers and students agreed that skills 
were important, they felt that the issues and problems faced by students in 
their lives at work and elsewhere was a better starting point. They felt that a 
curriculum based only or primarily on de-contextualized skills was too 
narrow for this setting. (It is interesting to note that the program at Kennedy 
Die Castings, Inc. had come to the same conclusion.) 

From these needs, a curriculum-as-process approach was used; curriculum 
was built in the classroom, from learners* needs and organized around themes, 
or "commonalities in the experience of the learners." (This is described in more 
detail in section C.) 
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In this definition, -curriculum" has a more European meaning. It is not only a set 
of competencies or skills, although these are part of it. It consists of these, and 
instruction, and teachers, and students, and the classroom; the curriculum is the 
program and the process through which learners learn. Also, the definition of 
"curriculum" and every other aspect of the program is touched by a "learner- 
centered" or "participatory" philosophy, which Johan Uvtn summed up when he 
said that one of the program's goals was to "exhaust the level of worker 
participation, given the constraints of the grant and the nursing home." 

Summary of Curriculum and Curriculum Development Across the Four 
Programs 

The range of curriculum definitions and models represented by these four 
programs is great. At Hampden Paper Company and South Cove Manor 
Nursing Home, curriculum is more often created in the classroom in response to 
student-articulated needs. In both programs, however, this requires a great 
deal of advance needs assessment and planning. At Boston City Hospital and 
Kennedy Die Castings, Inc. a set of written competencies is developed, taught, 
and revised on an ongoing basis. Of course, these are adapted to particular 
learners in particular classes. 

In all four models, in different ways, both student/workers' needs, as determined 
by students, and the need for better-skilled workers, as determined by 
management (and labor) are addressed in one way or another. In no case are 
either student or employer needs ignored in the curriculum or curriculum 
development model. And in cases where organized labor is a partner, at 
Boston City Hospital and Hampden Paper Company for example, their 
perception of needs is also included. 



B. Curriculum Goals: Range and "Central Tendency" 

We asked coordinators, teachers, evaluation teams and, where possible, 
students what the goals of the program curriculum were. We attempted to 
determine how clear various participants were about the curriculum goals, and 
if there were any curriculum goals which were held by several programs. 
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We found that there were several goals which all four programs held; they 
wanted their curriculum to help workers: 

1. Perform their jobs better; 

2. Communicate better with supervisors, patients/residents, and/or co-workers 
in English; and 

3. Gain new opportunities to advance up the career ladder. 

We also found that there was a very great range of goals, some of which were 
unique to a program. For example, at Boston City Hospital, one curriculum goal 
was to provide the hospital management with people who are read/ to be 
trained as technicians in order to address a labor shortage. At South Cove 
Manor Nursing Home one curriculum goal was to help student/workers better 
prepare to respond to federal requirements for certification for nursing 
assistants. Another goal of this program, as well as the Kennedy Die Castings, 
Inc. program, is improved safety. At Hampden Paper Company a goal of the 
curriculum (which although not articulated, also might be shared by other 
programs) was to increase decision-making capacity for workers so that they 
could use skills which in the past were left to management. At Boston City 
Hospital and at Hampden Paper Company GED attainment was a curriculum 
goal for student/workers. Two programs, Kennedy and South Cove Manor, also 
had as a goal that the curriculum and program should increase employee 
loyalty and/or retention. 

C. Curriculum Development Processes at the Four Programs 

In section A we touched upon the process of curriculum development used at 
each program. It will be described more completely in this section. 

All four programs have ongoing curriculum development processes. In some 
cases, for example at Kennedy Die Castings and Hampden Paper Company 
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the summer is a time for major curriculum work. There are many reasons why 
frequent curriculum revisions are necessary: 

1 . The workplace is dynamic. As work needs change and job tasks and 
descriptions change, a responsive workplace education program must 
change to meet these needs. Workplaces undergoing changes in 
technology, especially computerization, or workplaces affected by economic 
forces may experience major changes, and the curriculum for such a 
workplace education program may need frequent major overhaul; 

2. These programs are new, and the curricula which were created or adapted 
for a specific site, are often rough and need to be refined; 

3. All of these programs want to be responsive to a wide range of student 
needs. While some needs are currently addressed by parts of their 
curriculum, other needs are not. Also, although usually one person usually 
has the greatest responsibility for curriculum development at each program, 
other teachers, supervisors, advisory board members, and students are 
involved to varying degrees in the process at all of the programs. 

Kennedy Die Castings. Inc. 

The Worcester Workplace Education Collaborative has structured the time of 
Kathy Soderstrom, the full-time Head Teacher/Curriculum Developer, so that a 
major part of her job is curriculum development, up to 10-15 hours a week. 
Other teachers in the collaborative spend up to an average of two hours per 
week. This time is not a luxury; it is a necessity if the curriculum is to be tailored 
to student and company needs. Some of it is spent in direct observation on the 
shop floor, some in writing, revising, and trying out new materials. 

To varying degrees student/workers, teachers, the company Vice President, and 
supervisors all contribute to curriculum development. Student/workers do goal- 
setting in class. Supervisors review and add to these goais in the classroom, 
and informally discuss workers' language needs one-on-one with the teachers. 
Robert Kennedy, the Company Vice President, has been involved to represent 
company needs; for example, to see that the curriculum addresses basic skills 
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needed for the new quality assurance process. Ken Carson, the former 
Management Representative on the advisory committee, was ciosely involved 
in identifying needs, and from time to time, in presenting information to classes. 

Boston City Hospital Curri culum Development Process 

The current curriculum, created by Paul Guldenzoph, is the "daughter of a 
curriculum written by Merle Coughlin and Mary Jane Schmitt who were 
teachers at the Community Learning Center in Cambridge. This curriculum 
seemed especially germane as it had been designed to prepare minorities for 
allied health careers. 

Other sources of curriculum change were a City Roots (a GED preparation 
program for Boston school dropouts) curriculum, and materials suggested by 
the librarian at the Adult Literacy Resource institute, James McCultough. tn 
previous years, Paul said he had also been able to spend some time in the 
student/employees' work environment, and collected actual documents they 
needed to use at work, for example the Patient Care Conference form. Also, 
overtime, Boston City Hospital teachers added to materials and competencies. 
In the first two years (when the program had other teachers) there were monthly 
meetings at which the curriculum would be reviewed, but Paul felt that more 
productive changes occurred as a result of a teacher pointing out a particular 
problem a student/employee was having. For example, the "Managing Your 
Manager series in the curriculum resulted from such an informal needs 
assessment. 

Although in past years the curriculum was reviewed by the workplace education 
program advisory committee, neither this committee nor the evaluation team 
had input into curriculum development this past year. 

Hampden Paper Compan y Curriculum Development Process 

When the program started, according to company President, Robert Fowler, 
they deliberately kept job-specific skills out of the curriculum. They wanted 
workers to see the program as providing life skills which would meet their own 
needs; but it soon became clear that employee participants wanted to have job- 



related skills; they wanted the workplace education program to be relevant to 
their basic skills needs at work as well as their other basic skills needs. As a 
result, the curriculum includes such diverse areas as "reading a road map,", 
"getting a GED," "learning accounting math to use in keeping books for a small 
business," "reading a grid or political map," and "chemical mixing." 

The process for curriculum development is evolving, and has several different 
parts. Most of the curriculum development is in three areas: the "skills 
checklist," (i.e. the format students use to select competencies for their 
individual learning plan); job bidding (e.g. formula ticket preparation); and 
computer tools learning. All three areas evolve from needs determined by 
student/workers and supervisors. Curriculum development is done primarily by 
the Teacher/Coordinator, Janice Rogers and with assistance from the Program 
Director, Alexis Johnson, and input from the Advisory Committee. 

Curricula are also adopted (and adapted) from other sources, e.g. the Jobs For 
Youth-Boston GED preparation curriculum, a math curriculum developed by 
Mary Jane Schmitt, and a process writing curriculum. 

An exciting aspect of curriculum development at Hampden Paper Company is 
the taiior-made lessons which grow from supervisor and student-defined needs. 
Don Mirable, a supervisor who sits on the evaluation team, commented that he 
felt that tailor-made curriculum development (e.g. the chemical mixing formula 
ticket preparation) especially opens doors for employee advancement. This will 
be discussed further in section I. 

South Cove Manor Curriculum Process 

As described earlier, the curriculum development process at South Cove Manor 
Nursing Home was changed dramatically. The following describes the current 
collaborative model which was in place during the curriculum study. 

Everyone affected by the workplace education program at South Cove Manor is 
involved in a remarkably collaborative curriculum development process: views 
are sought on the needs of learners from learners themselves, from their 
teachers, from the nursing home residents who receive their services, from 
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those who are responsible for training and supervising learners, and from those 
who are responsible for the overall management of the nursing home. Much of 
this takes place through individual and small group meetings. 

Johan Uvin, the On-site Coordinator/Instructor, and his colleagues assume that 
student/workers have the potential to identify what they need, and to go about 
meeting their needs, that with help from teachers they can facilitate their own 
needs assessment, set goals, and be involved in making their own decisions 
about how to learn. 

Working from this assumption, needs of learners are elicited in classes as 
learners participate in "decid(ing) what need(s) to be learned, how, and why, 
and in choosing ways to document learning as it (takes) place" 
(Caldwell/Change/Uvin, 91, Chapter 4). Yet, if teachers ask too directly "What 
do you want to learn?" or "How do you want to learn?" students often can not 
answer with much sophistication. Instead, teachers help students to research 
their learning needs, methods and styles. This process often begins with 
questions such as: "What are you doing in your work? Where are you using 
English? What problems are you having that you could solve if you knew more 
English?" In the one-on-one planning sessions with students, these questions 
are the starting point for their research. 

Answers to these questions are discussed in class and experiences are shared. 
There is an effort to determine if other student/workers have had a particular 
experience at work, if others share a particular problem, issue, or concern. With 
this information, teachers look for commonalities, for themes which grow from 
learners' experiences, themes which can be analyzed, and which will lead to 
new skills in language learning, new cultural and cross-cultural knowledge and 
attitudes, and learning about the work environment. The teachers encourage 
students to discuss their individual experiences, to learn if other students have 
experienced this, and have similar concerns or issues. 

Once it is clear that a certain set of experiences and issues are common to 
students in a class, the theme is named, and specific examples of the theme, 
from the context of students' experiences are described. 



ERIC 



From the examples, which are rooted in the shared experiences and concerns 
of the learners, specific learning activities are designed: role-plays, dialogues, 
and others. These address the theme , but also include specific language skills 
to be learned, and often cultural and work-related knowledge and skills. 

Students engage in these learning activities in the classroom, and as they are 
learning new-and contextualized-language skills, they are also formulating 
answers to their questions about the social, political, and cultural contexts in 
which they are working, and to problems at work and in their lives outside work. 

The last and perhaps most difficult step of the process, is documenting the 
learning outcomes from the activities in relation to the overall articulated 
program goals. 

This process is not linear or deductive. It can be entered at any of these steps. 
For example, often the starting point for a lesson is a problematic incident at 
work described in class or in a meeting between a student and a teacher, 
supervisor or administrator. Once the process is entered, however, the cycle or 
spiral of steps is usually followed. 



D. Where Curriculum Objectives and Contents Come from 

One would expect the contents of a workplace education program curriculum to 
come from needs at work, but the Workplace Education Initiative state partners 
also expressed interest in knowing to what degree curriculum content is derived 
from student needs outside work. 

We broke down the various sources of curriculum objectives and content into 
these two categories: 1 ) learning needs from the work environment; and 2) 
learning needs from outside work. These were further broken down into the 
sub-categories which appear below. From our interviews and from direct 
observation, we found that nearly all the programs had curricula which 
attempted to meet the needs in most of these sub-categories, and that all 
programs addressed both work-related basic skills needs and learning needs 
from other parts of student/workers' lives. 
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Sources of Curriculum Goals and Content 



Following each sub-category below, we indicate how many programs have 
curriculum content which addresses needs from that category. 



1. Learning Needs from the Work Environment Which Are Addressed in the 
Curriculum 



a. Language/Grammar Needs at Work: All programs 



b. Other Basic Skills at Work: All programs 



Examples of basic skills content in the curriculum which come from the 
work environment included: reading; writing; oral communication skills; 
math (measurement, and math for quality control procedures); cross- 
cultural skills and understanding; note-taking, decision-making, 
analytical thinking in English, organizing one's learning; and 
understanding patients' rights and workers rights. 



c. Issues at Work: All programs 



Examples of kinds of issues included: dealing with supervisors and co- 
workers; changes in the company; work slowdowns; quality circles and 
self-inspecting; getting along with people from other cultures; child care; 
and conflict resolution. 



d. Labor/Union Issues: Three programs 



Because Kennedy Die Castings, Inc. does not have a union, they did not 
identify union/labor issues. At South Cove Manor, however, Johan Uvin 
felt that the curriculum included labor issues even though the workplace 
is not unionized. Boston City Hospital and Hampden Paper Company 
included labor/union issues such as: reading the contract; reading and 
writing grievances; and speaking up at meetings. 



98 llo 



e. Student/Worker Empowerment: All programs 

As might be expected, "empowerment" was defined in different ways by 
different programs. At Kennedy Die Castings, Inc., for example, it was 
defined as being effective, speaking up, having self-confidence, solving 
problems, and increasing one's personal power. 

2. Learning Needs From Outside Work Which Are Addressed in the Curriculum 

a. Basic Skills Needs in Workers' Lives Outside Work: All programs to 
varying degrees 

We gave some examples of this: reading and helping with children's' 
schoolwork; writing notes to a child's teacher; reading to children; 
reading the Bible; and doing income taxes. Kennedy Die Castings, Inc. 
education staff said that for their program this kind of learning was not 
emphasized, and was done on an ad hoc basic. At South Cove Manor, 
non work-related basic skills are more likely to be addressed in tutoring 
sessions, but they may also be addressed in class as students bring their 
needs to class. One example of this was a student who brought an 
official-looking letter to class which he had received from what he thought 
was a government agency. He needed to know what this letter was, and 
what response was required. (It was actually an unsolicited letter from a 
singles matching service!) Another example was a class field trip of 
beginning ESL student/workers to the Boston public library to get 
children's' books in English and books in Chinese for adults to read. 

b. Language/Grammar Needs Outside Work: Three programs 

c. Life Skills: All programs, at least to some extent, often integrated with 
work-related skills 

d. Individual Growth: Three programs 

South Cove Manor includes this in individual conferences, and Kennedy 
Die Castings, Inc. and Hampden Paper Company interpret this as part of 
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the confidence- and self-esteem building that is woven into the learning 
activities in their programs. 

e. Family Issues: All programs 

This meant different things in each of the programs. For example, at 
South Cove Manor, family planning has been a major theme. At 
Kennedy Die Castings, Inc., holidays and customs, setting up a doctor's 
appointment, and immigration issues have been included. 

f. Community Issues: All programs 

Here, too, the issues varied in the four programs, but included: 
developing a community resource guide: neighborhood safety; children 
and child care issues; and registering to vote; among others. 

g. Other Issues Mentioned 

At Boston City Hospital career and educational planning are issues that 
are addressed in the curriculum in a formal way. This issue is addressed 
at other programs in various ways.. 

The Kennedy Die Castings, Inc. program staff pointed out that because of 
federal funding requirements, they do not formally include non work- 
related learning in their curriculum, that these needs are addressed, but 
informally and not nearly to the extent of work-related needs. The 
Administrator at South Cove Manor expressed a strong concern that 
everything in the curriculum in their program must be work-related, a 
view which was not shared to the same extent by the program staff 
interviewed. The Vice President at Kennedy, Inc. said that the curriculum 
must be work-related, but that this must not be defined too narrowly. He 
said that teaching an employee to read so he could read his medical bills 
and pay stubs, and teaching him to speak English well enough to deal 
with the staff at a medical health plan were both work-related as far as he 
was concerned. Of all four programs, Hampden Paper Company 
seemed least constrained in their attempts to address non work-related 
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learning; but because both management and workers preferred it, they 
were moving to include more, not less, work-related basic skills learning. 



E. Skills Addressed in the Curriculum 

We wanted to know what basic and other skills were addressed in the curricula 
of the four programs. We found a great deal of consistency across the four 
programs. Table A in the Appendix: "Skills Addressed in the Curriculum 1 ', gives 
a detailed program-by-program breakdown of the information summarized 
below. 

The following basic English language skills were included in the curricula of all 
four programs: listening, speaking, reading, and writing. (Speaking was 
included, but not a priority at Hampden Paper Company and writing was 
included, but not a priority in the curriculum at South Cove Manor.) All four 
programs included grammar in the curriculum, and especially in the two ESL 
programs, grammar was integrated into the topics and themes. 

All four programs included some computation skills, but as far as we could see, 
no program had a comprehensive, systematic approach. At Kennedy Die 
Castings, Inc. and at South Cove Manor Nursing home, computation skills were 
not identified as a need to be addressed. 

All four programs included life skills learning, but these were not a priority at 
Kennedy Die Castings, Inc. , at South Cove Manor Nursing Home or at Boston 
City Hospital. 

All four programs included: the ability to work independently; critical thinking 
and problem solving skills; and learning about others' cultures. 

Teamwork/cooperation was included in the curriculum at Kennedy and Boston 
City Hospital 

Management skills were included to some extent in two programs: Kennedy 
and Boston City Hospital 
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Secondary level subject areas were included only at Hampden Paper 
Company and Boston City Hospital, where students can prepare for the GED. 

in response to a question about other skills which the curricula included, we 
received the following answers: 

Kennedy: Co-operative learning skills 

South Cove Manor. Metacognitive skills, such as learning what kind of learner 
one is, by bein& exposed to different kinds of learning activities and reflecting 
on their efficacy. 

Boston City Hospital: Note-taking skills, test-taking skills, and career and life 
planning skills. 

F. Participants in Curriculum Decision Making: Students, Supervisors.and 
Others 

Who participates in curriculum decisions, and how, varies across the four 
programs. 

Kanneriv Die Castings. Inc. 

There is no formal process here for students, as a group, to participate in 
curriculum decision making; but collaborative student goal-setting in class has 
resulted in curriculum changes. And students who we interviewed felt that they 
had had an opportunity to set goals in the classroom, individually and as a 
group. 

Supervisors participated in the class goal-setting process, too, when they 
attended a class early in the spring cycle to comment on, critique and add to the 
learning goals students had generated. Below are the results of that goal- 
setting session: 
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The two questions which prompted the list of goals below were: "Why do we 
need English?" and "What do you need to learn to do your job better?" The 
following were some of the students' responses and supervisors' additions. 
(Those with an * were added by supervisors.) 

• Drive to work (read signs) 

• (Get U.S.) citizenship 

• Speak with supervisor, co-workers, nurse, Ken Carson (Human Resources 
person) 

• Understand paycheck 

• (Talk with) doctors and hospital (personnel) 

• Read and write on forms, (do) "paperwork" 

• * (attend, understand, benefit from) training programs at Kennedy 

• Identify quality defects: 

- Identify actual defects and complete the paperwork to record these 
(reading and writing) 

a. What are they? 

b. Tell the supervisor/quality control. Speak and understand English. 

c. Speak with co-workers. 

• Safety 

- read signs 

- give warnings 

- heed warnings 

" Understand your supervisor 

• 'Problem solving 

• 'Understand simple directions (especially in cross training) 

The list of goals which was produced by students was not significantly changed 
by supervisors; perhaps this is because quality assurance changes in the 
company were driving this list for both workers and supervisors. 

Robert Kennedy, the company Vice President, has reviewed curriculum, and 
given it overall direction. He pointed out that Ken Carson, the management 
representative on the advisory group, has been involved in developing some 
curriculum units (e.g. "Aluminum Die Casting at Kennedy"), and the guest 
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speaker component. Robert Kennedy, too, occasionally teaches a class on die 
casting principles and defects, and he has narrated a slide show for the class. 

From Robert Kennedy's point of view, the needs of the curriculum come in large 
part from the needs of the company, and the company must be responsive to 
the changing needs and demands of customers and the industry. Curriculum 
decision-making is affected by these. For example, now the company requires 
statistical process sheets as part of quality assurance; these require writing 
skills. Machine operators are now responsible for quality, not a quality control 
inspector; in order to do this they have to be able to write in English to fill out 
tags. And, as Robert Kennedy put it, " Increasingly workers have to understand 
the process as well as complete the task." These changing demands create 
needed changes in the curriculum. 

Boston City Hospital 

There is no formal involvement of students in curriculum decision making at 
Boston City Hospital, but Paul Gufdenzoph says that the problems they present 
to him in class, and in individual meetings, are the sources of changes he 
makes in the curriculum. Paul also pointed out that their annual evaluation of 
the courses leads to curriculum changes/Paul said that supervisors are 
involved informally, without necessarily knowing that their comments have been 
a source of curriculum change. 

South Cove Manor Nursing Home 

By ccitrast, at South Cove Manor Nursing Home, students have a great deal of 
choice, both individually and as a group. The curriculum is formed from their 
articulated needs for English language skills in work and other life contexts. 
Most of the needs they bring appear to be work-related, although students 
mentioned language needs related to greeting and interacting with American 
friends, cooking, asking for directions on the street, and using the subway as 
examples of non work-related language needs. 
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Hampden Paper Homnanv 



At Hampden Paper, individual students determine what they will study. With the 
exception of the GED curriculum (which has its own set of objectives because it 
is designed to prepare students for a prescribed test), an individual curriculum 
is generated for each student. For example, students can work on comparison 
shopping, tax forms, balancing checkbooks, writing letters, as well as preparing 
for job bids, and learning to do word processing for work-related or other needs. 

Students also have a great deal of say about what learning materials they use: 
books, workbooks, formula tickets, regular and adult new reader newspapers, 
the computer, maps, etc. 

At Hampden Paper Company supervisors complete a feedback form, 
sometimes orally, sometime in writing, on program attendees from their 
departments. Sometimes this information has an impact on the curriculum. 
Supervisors also participated in the development of the recent skills checklist, 
as did the advisory committee. 

G. The Instructional Setting 

The sections on instructional methods, activities and materials which follow can 
be better understood in the context of each of the four different physical 
instructional settings. For this reason, these settings are briefly described below. 

Kennedy Die Castings. Inc. 

if you visited the beginning level Workplace ESL class at Kennedy Die 
Castings, inc. you might see a predominantly male group of 8-10 Southeast 
Asian, Hispanic and Polish student/work&rs in the company lunchroom known 
as the cafeteria. A few employees who were not students might come in and 
out to rest, use the food vending machines, or to learn by watching and listening 
from the sidelines. You would hear the occasional noise of the vending 
machines, and the constant background noise of machines on the shop floor 
outside. You would see tables and chairs arranged in a "U" shape in a 
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comfortably large space, facing a felt board and perhaps teacher, Kathy 
Soderstrom. Although the room doesnt establish a sense of belonging to the 
class, it Is functional, cheerful, comfortable enough, and makes the program 
seem very much a part of the work environment. 

The day we visited the class we observed Kathy doing a learning experience 
chart on weather, saw the class reading a weather forecast from the Worcester 
Telegram and Gazette, listened to questions (from Kathy) and answers (from 
students) about holidays and days of the week. We also saw diads of students 
asking and answering weather-related questionc, and Kathy circulating to help 
them. There was some writing activity-filling in the blanks- followed by reading 
out loud, and conversation about weather pattern language. The lesson 
included embedded language skills on the use of the present, past, and future 
of the verb "to be." 

Boston City Hospital 

Classes are held in comfortable, well-lit, spacious college-style classrooms in 
Boston City Hospital's training wing. The classroom has a teacher's desk up 
front, and plenty of chalkboard space to write on. In the pre-technical science 
and study skills class size there were 12 students enrolled, with 8 attending the 
day we visited. In the ABE class only 4 students attended when we observed 
the class. The classroom, used by both classes, was admirable in every way 
except its temperature. During one visit it was too warm; during the other much 
too cold. 

The day we visited the Science/study skills class, we saw group instruction, 
primarily lecture, delivered college-style from lecture notes, with chalkboard 
diagrams, and discussion. Students were practicing taking notes, and 
occasionally referring to textbooks. There were few teacher-prepared handouts 
in this class. As the purpose of this class is to prepare student/workers for post- 
secondary training programs, where there will be lecture, note-taking and 
textbook reading, perhaps what we saw accurately simulated what * h ey will 
experience; however, the teacher, Paul Guidenzoph, saw his purpose as 
helping to reveal the mystery of this learning model by breaking down new 
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information into small, digestible pieces, and by encouraging and respectfully 
answering questions, a style rarely found at the college level. 

In addition to the classroom, Paul Guldenzoph has a two-room office space 
nearby where he can tutor and counsel students and store instructional 
materials and supplies. 

Hampden Paoar Comnanv 

Hampden Paper is proud of the space it has made available in the factory for its 
classroom. In this comfortable, quiet space, tables and chairs are arranged so 
that students can work independently, under teacher supervision, or work one- 
on-one with a teacher or colleague. There are two computers, plenty of wall 
space for maps, charts, and other instructional media, and there is adequate 
space for keeping instructional materials. Unfortunately, there is no private 
space for counseling, but the audio-visual room next to the classroom can be 
used for this purpose. 

Student/workers attend the center by appointment as their schedules permit, 
usually up to 5 hours weekly. Since the fall of 1990, each student has had a 
"skills checklist", a sort of individual education plan. This list serves as a 
planning tool, but also as a way to record what students are studying and what 
they have learned. Students doing GED preparation use the student record 
included in the Jobs For Youth GED curriculum. There is a very wide range of 
skill levels in the center: of the 1 1 students who visited while we observed, 2 
were ESI, 3 ABE, 4 GED prep., and 2 Post-GED. 

South Cove Manor Nursing Home 

South Cove Manor has two rooms available for classes: a small, well-lighted 
physical therapy room and an administrative conference room. The conference 
room, on the first floor is quiet, but very public; its glass walls face the street and 
the nursing home reception area Both rooms are rather small, especially for a 
class of ten students. 
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The physical therapy ;oom offers privacy, has a storage cabinet for learning 
materials, and a small, portable, erasable marker board. (At one time it also had 
a chalkboard.) It is reasonably quiet, but some noise drifts in from the hallway 
where residents or staff play music, sing, or converse. During both visits we 
found this room rather warm, but no one else seemed uncomfortable. 



H. Instructional Methods and Activities 

We wanted to know what instructional methods and activities were used in the 
classrooms at the four programs, and how consistent this was across programs. 
We also wanted to know if a variety of methods and activities was used at each 
program. We interviewed teachers and/or teacher coordinators and students, 
and we observed classes at each program. The information from our 
observations was consistent with the interviews. For Breakdowns of these data 
by program see Table B: "Instructional Methods and Activities" in the appendix. 
Surprisingly, only three methods or activities were used at all four programs: 
vocabulary developed from on-site responsibilities; case studies; and drill and 
practice. 

However, several methods/activities were used at three of the four programs, 
including: discussion (Hampden Paper formerly used discussion when students 
met in groups); simulations or role-plays; teacher modelling of skills for the 
class; tutoring; and viewing videotapes. 

A few methods/activities were used at half of the programs: co-operative 
learning activities; dialog journals; and picture stories. 

The following were used by at least one program: presentation/lecture (a major 
method at Boston City Hospital); reading comprehension derived from family 
stories; choral work; games; reading articles, charts, etc. from newspapers; 
discussing tools and defective part samples; language experience charts; 
computer tools, such as word processing and spreadsheets; and workbooks. 

Each of the four programs used a variety of methods and activities. From 
observation of a class, and from the curriculum, it was clear that the Kennedy 
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Die Castings Program uses a very wide variety. Although the two classes we 
saw at South Cove Manor did not show evidence of it, it was clear from 
curriculum materials we saw, and from interviews, that South Cove, too, uses a 
very wide variety of methods and activities. 



I. Instructional Materials 

We were interested to know if teachers used both "home-grown* and 
commercial materials, and both print and non-print (including tactile) materials. 
We were also interested to see the quality of teacher-made materials. And we 
wanted to know how commercial materials were screened and by whom. 

In all four programs both teacher-made (or teacher-adapted) and commercial 
materials were used, although the degree varied greatly in the four programs. 
And in all four programs both print and non-print materials were in evidence. 
Non-print materials which we saw in the classroom or were told about included; 
drawings, diagrams, cuisenaire rods, charts and graphs from newspapers, 
computers, slide shows (the Kennedy company tour slide show) videotapes, 
computer software, work tools, safety equipment, and defective products. 

We did not get a clear picture of how commercial materials are screened. As far 
as we could determine no program had a formal screening process, but left this 
to the education program staff on an ad hoc basis. Yet, the materials we looked 
at seemed free of notable cultural, racial, and gender bias, and were written for 
adults, not children. Screening does not appear to us to be a problem. 

At Kennedy Die Castings, teacher-made (actually curriculum developer and 
teacher-made) materials are extensive, and are well documented. Many are 
inspired by "English at Work - A Toolkit," a model which emphasizes using 
realia from the workplace and themes of student interest and concern. A 
relatively complete workplace English curriculum was specifically designed 
from scratch for Kennedy Die Castings. It is a remarkable, if not yet refined, 
curriculum product, and a model for other programs in the initiative to emulate. 
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At South Cove Manor, too, most of the instructional materials are teacher-made. 
Two curriculum products have been produced: a competency-based resource 
manual; and, In collaboration with the Continuing Education Institute, of 
Needham, a curriculum guide for nursing assistants, which will be published 
later this year. 

Hampden Paper Company 

A very creative and successful "chemical mixer formula ticket" curriculum unit 
was developed at Hampden Paper by substitute teacher, Margaret Lobenstine. 
The process and the product are together an exciting example of an original 
and significant curriculum materials development effort. This kind of work, we 
must note however, requires a significant amount of a teacher's time. Below is 
a brief description of the process Margaret Lobenstine used to develop this unit. 
Copies of some of the materials are also included in Appendix C. 

The Formula Ticket Lessons: 

A set of "Chemical Mixer Formula Ticket* lessons was developed for one 
student/worker, a person with limited English proficiency, who wanted to be 
able to successfully read and correctly cany out the procedures for mixing 
certain chemicals. 

The information a worker needs in order to perform this chemical mixing task is 
contained on a single-page "Formula ticket". (See Appendix C for example). 
The information is succinct and contained within various areas of the form in an 
order which apparently makes sense to the chemists who make out the formula 
tickets, but which is not the order in which the steps of the process are carried 
out by a production worker. Because it assumes thorough knowledge of the 
process, and is a set of job specifications rather than a sat of instructions for 
carrying out the process, the Formula Ticket is difficult to read, even for a native 
speaker of English. For someone who has no knowledge of the process to 
which it refers, it would make little or no sense. 

The process used to develop this "mini-curriculum" on the Chemical 
Mixer Formula Ticket was as follows: 
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1 . The teacher visited the work area, got a copy of the "Formula Ticker and did 
a step-by-step analysis of the chemical mixing task. The task analysis was 
reviewed for accuracy by the work supervisor. 

2. The teacher interviewed the student to determine if there was an area of his 
experience which was analogous to chemical mixing. In this case, there 
was. The student was an experienced wine-maker, and wine making is a 
process which requires that certain steps be done in a prescribed order and 
with relative precision in mixing amounts. The teacher asked the student to 
explain to her, to teach her, how to make wine. Together, teacher and 
student broke into small steps the directions he gave her. They also 
discussed what the consequences would be if he didn't pay attention to 
exact amounts. 

3. Then the teacher listed the steps in following a Chemical Mixer Formula 
Ticket, and as she went over them, with an actual formula ticket in hand, she 
drew upon the analogy of his experience with wine-making. As they did this, 
they focused on the logic of the procedure, the critical understanding of the 
process, not rote memorization of procedures. She asked, for example, 
"What would happen if you let the mix sit for half an hour before filling in the 
temperature/viscosity ticket instead of after?" Also, she introduced new 
vocabulary words as they were used in the process. 



J. Staffing, Staff Orientation and Staff Development 
Staffing: 

Staffing patterns at the four programs differ greatly. This is based in part on 
whether the program is a single site or part of a multi-site program, how many 
classes are offered, and funding levels, among other factors. 

Kennedy Die Castings. Inn 

The Worcester Workplace Education Collaborative has several workplace 
education sites. There is a Project Coordinator for the Collaborative, and 
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instructors at each site. At Kennedy, the instructor is also the Head 
Teacher/Curriculum Developer for all the sites. 

Rnstnn City Hospital 

BCH has one paid, full-time staff person; he is the Teacher/Counselor and 
Program Coordinator. His responsibilities include administration, teaching, 
curriculum development, counseling, coordinating and acting as a liaison 
between employees, supervisors, and upper level administrators. There is also 
a part-time volunteer math tutor. 

Hampdfln Paper Comnanv 

Hampden Paper has two staff people, the Program Coordinator from 
International Language Institute of Massachusetts (4 hours per week), and a 
Teacher/Coordinator on site (25 hours per week). (During the period of this 
evaluation, because Janice Rogers, the Teacher/Coordinator had been on 
leave, we also interviewed her substitute, Margaret Lobenstine.) The 
Teacher/Coordinator is on site from 12-5 PM, every week day. 

The Teacher/Coordinator teaches, assesses (with TABE pre- and post-tests as 
well as ongoing assessment), and develops the curriculum and instructional 
materials (which often involves starting from scratch and drawing on what's 
available to meet student-articulated needs). 

Soirth Cove Manor Nursing Home 

South Cove Manor has three staff: a Project Coordinator (3.5 hours/week), a 
full-time On-site Coordinator/Instructor, and a part-time Instructor (30 
hours/week). This staffing level allows 5 classes per week as compared, for 
example, with one class per week at Kennedy Die Castings, Inc. It also allows 
for needed curriculum development time. 
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Staff Orientation and Staff Development: 

We asked about staff orientation and staff development activities at two of the 
programs we visited. From these interviews and from written materials we 
learned the following: 

Kennedy Pie Castings, inc. 

At Kennedy, teachers are oriented using the Quinsigamond Community College 
Adult basic education teacher orientation, followed by more specific orientation 
activities for workplace education teachers, including: a visit to the workplace; 
and from photographs of the workplace, students describing their jobs and the 
workplace to the new teacher. 

Most staff development activities are not planned out, but are available as 
opportunities arise. 20 paid hours a year per instructor slot are available for 
outside professional development. This is used primarily for SABES (Mass. 
System for Adult Basic Education Support) staff development activities, and 
conferences, and is part of the teacher's contract. In addition, orientation and 
summer staff curriculum development are also paid staff development time, and 
staff are encouraged to network with staff from other adult education programs. 

Teachers are paid for two hours per week of staff preparation time and they are 
paid for required meetings. One- or two-hour staff meetings are held each 
month to review curriculum, discuss workplace education issues, network, 
problem solve, share ideas, and collaborate to write curriculum. 

Hampden Paper 

Staff preparation time is paid for and curriculum development time is included in 
this. 

There is no formal process for staff orientation. When Janice Rogers, the on-site 
Teacher/Coordinator, was hired there were orientation meetings and a 
company tour at her request. When a long-term substitute teacher was needed, 
for example, there was a 2-3 week overlap so she could learn the ropes with 
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Janice's assistance. Also, There is paid staff development time to attend in-state 
and out-of-state professional conferences, in-house (international Language 
Institute) workshops and SABES workshops. 

South Cove Manor Nursing Home 

The most developed staff orientation and development plan of the four 
programs is that at South Cove Manor. The 12 hours of Staff orientation there 
includes: an orientation to and tour of the facility; working one eight-hour shift; 
meeting informally with program participants; learning about the goals, structure 
and patient care delivery processes; learning about the history, philosophy, and 
goals of the program and the relationship to South Cove Manor goals and to the 
agenda of the National Workplace Literacy Movement. 

Following orientation, staff participate in 6 hours of training for the job. Training 
includes an introduction to the functional context approach; an introduction to 
the program's initial and current approaches to curriculum development as well 
as other approaches; information about human, physical, and materials 
resources, work-related policies, and record-keeping and curriculum 
documentation procedures. They also have 10 hours of on-the-job training 
including: meeting with and observing a peer teacher, team teaching, meeting 
with supervisory/licensed staff, taking the national nursing assistant 
examination, and other activities. 

In addition, teachers participate in a number of ongoing development activities, 
including: feedback sessions, advisory committee meetings, meetings with 
nursing home personnel, monthly peer observation, mentor coaching, monthly 
observation by the coordinator, weekly meetings of all program staff, monthly 
workplace education teacher sharing sessions at the Adult Literacy Resource 
Institute, professional development projects, conferences and workshops, and 
other activities. 
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K. Impact of the Curriculum on the Workplace Culture 

We were interested to know if participants felt that the curriculum had an impact 
on the company culture , its practices, policies, and/or priorities. This is very 
difficult to know precisely, but in response to our questions we received a 
generally positive response, indicating that most of the local level partners felt, 
at least, that there was such an impact. 

At Kennedy Die Castings, for example. Ken Carson, a member of the evaluation 
team, pointed out that the curriculum has had an impact on safety. As an 
example, he suggested that if six years ago they had asked workers to wear 
earplugs, the workers wouldnl have understood. Now, he said, this can be 
explained in English. 

Company Vice President, Robert Kennedy, said that the company lunchroom 
used to be segregated. Joking took place only in employees' native languages. 
Now, he says, there is more comraderia.in English, across cultures. The 
Vietnamese sit with the Italians; men with women. Now people look at what 
they have in common. There is mors social, non work-related use of English. 
He attributes this to the program and its curriculum. 

The evaluation team members felt that because the class is held in the 
cafeteria, a very public place, it is well-known, and is seen as part of the 
company culture, helping workers and helping the company. 

All members of the evaluation team felt that worker self-confidence in using 
English had increased as a result of classes. Of course, it is hard to know what 
part the curriculum has played in this. 

The education staff at Kennedy Die Castings, Inc. program pointed out that 
workplace education tits well with company goals, especially for increased 
interactivity among employees, that company culture is being affected in a 
direction in which the company administration wished it to move. They believe 
that the curriculum has also helped to change majority culture employees' 
perceptions of linguistic minorities, that it has resulted in increased sensitivity to 
the difficulties of immigrants who do not have sufficient English language skills 
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needed to solve problems; they believe the curriculum (as part of the program ) 
helps to break stereotypes. They also believe that workers' perceptions of each 
others' cultures have been changed as a result of the curriculum. 

At Boston City Hospital, evaluation team members suggested that there is 
evidence of improvement in work-related skills on the units. For examples, 
employees better understand medical terms on the "Activities of Daily Living- 
sheets. They also thought that there was a change in employees' attitudes 
toward their education. Education (classes) is now part of the workplace, not an 
exception. (The SNAP program, they thought, has also helped to achieve this 
change.) 

At South Cove Manor Nursing Home, some evaluation team members 
observed that there is now less need to translate basic instructions such as how 
to take a temperature; that health workers have less difficulty filling out basic 
forms in English; and that there is increased social interaction between aides 
and residents, a friendlier atmosphere. 



L. The Multi-Level Classroom 

The multi-level classroom presents difficult problems for instruction, especially 
group instruction. We wanted to know if a program could offer classes at 
different levels and we discovered that there are several obstacles: 1) small 
programs often dont have enough worker/students for each level to warrant a 
single-level class, and they lack resources to provide enough teachers/teaching 
time to accommodate this; 2) if classes are offered during work time, it is often 
difficult to free up enough worker/students who need the class at a certain level 
at the same time; and 3) many worksites do not have enough physical space to 
offer more than one class at a time, further constraining the number of levels 
which could be offered. 

If the multi-level classroom is a given, and it probably is for most small 
workplace education programs, then we wondered what can be done to 
address individual students' needs within the multi-level classroom? 
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At Kennedy Die Castings, Inc., classes are multi-level and the curriculum is 
structured to accommodate this. For example, activities lend themselves to 
different levels, and within the activities students are often grouped according to 
ability, with the teacher, not the "curriculum" doing the levelling. Activities are 
more open-ended, not too structured, not too prescriptive, and can be easily 
adapted a varied to suit a particular class. 

At Boston City Hospital, there are two classes. One is at the post-secondary 
preparation level; the other is a multi-level class ranging from very basic literacy 
up through the GED. Tutoring is amply available outside of class, and this is 
especially useful for students reading and writing at the most basic level, and for 
others who need additional help. 

At Hampden Paper Company, one teacher addresses ESL (all levels), ABE (all 
levels), GED prep, and computer literacy. Everything is individually designed 
except the J FY GED curriculum. This is a huge challenge for that teacher, but it 
does address the problem of levelling. Each student is working at his or her 
own level. 

At South Cove Manor Nursing Home, there are two class levels: Beginner and 
Intermediate. The beginner class is 6 hours per week, and the intermediate 
class 2 hours per week. Still, within the beginner level especially, the classes 
are multi-level. The range of student needs is addressed through a theme- 
based curriculum, generated from the needs and interests of students, but with 
attempts by the teachers in the classrooms to offer several levels of skills. 
Although, in our brief observations, we did not see much evidence of different 
levels being addressed in either classroom, we believe that this does take place 
in some classes. 

We think that the approach used at Kennedy Die Castings, Inc. and especially 
at South Cove Manor Nursing Home, a theme-based approach based on 
language learning needs which emerge from experiences at wcrk, holds the 
most promise for the multi-level workplace ESL class. But as two students on 
the South Cove Manor evaluation committee pointed out when asked if they felt 
they had a choice over the curriculum objectives, when everyone in the class is 
at a different level, and when everyone has different needs, it's difficult to get 
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everyone's needs met. They thought the best solution was to focus on job- 
related learning, where there were more commonly-felt needs (and themes). 

Individualized tutoring, found at South Cove Manor, Boston City Hospital and at 
Hampden Paper, is also a strategy which offers some promise, although we did 
not examine this strategy in this study. 

Unfortunately, one of the most promising strategies in workplace education for 
addressing different levels, computer-assisted instruction, was not used by any 
of the programs. One explanation given for this is that the federal workplace 
education program discourages spending its dollars for capitol equipment such 
as computers. 

M. The Job Description for a Workplace Education Teacher 

The state level partners had asked us to determine what a workplace education 
teacher's job description should look like, including the skills, background and 
qualities needed, and the tasks of the job. 

To answer these questions we wanted to know what tasks teachers or 
teacher/coordinators currently did, and what experience, skills, and knowledge 
students, teachers, and evaluation team members thought were needed to do 
this job well. What follows is a synthesis of this information from these varied 
perspectives, and from all four programs. 

The Duties of a Workplace Education Teacher 

All of the following are currently duties of workplace education teachers or 
teacher/site coordinators: 

• Teach and/or tutor student/workers: ESL; beginning and intermediate level 
wo^-related basic skills; secondary level basic skills; computer literacy (using 
computer tools and feeling comfortable with the computer); etc. In some 
cases, where students ere learning English as a Second Language, teach 
using two or more languages. Use a variety of methods, activities and 



118 I3cj 



materials. Try to relate student learning styles to this variety. Use individually- 
paced and group-paced modes. Help students progress at their own rates. 
Use a learner-centered approach (as opposed to one which is teacher- 
centered, book centered or test-centered). 

• Create curriculum (including curriculum materials) 

• Review and select commercial curriculum materials 

• Recruit students 

• Assist with student intake into the program 

• Assess students' skills , interests, needs, and goals 

• Counsel students 

• Research the workplace: the nature of the work, the relation of the production 
or service parts to the whole organization, the relationships of various levels 
and kinds of personnel, the "corporate culture", the specific work each of the 
workers in the program does, the materials (especially written materials) used 
in the work. Get to know workers and supervisors on a first-name basis 

• Meet, network with, and collaborate with workplace education teachers in 
other sites or other programs 

• Learn from students, supervisors, union representatives, company 
management, and others 

• Assist students in learning about each others' cultures, about the "company 
culture", in the case of recent immigrants, about "American culture." Assist 
supervisors and other managers to understand student/workers' cultures 

• Publish (or assist students to publish) a program newsletter and produce other 
written materials and carry on other activities which make the workplace 
education program known within the organization 
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In the case of teacher/coordinators: 

• Administer or assist in administering a grant (record keeping, proposal writing, 
grant management) 

• Assist with coordination of local workplace education team for purposes of 
curriculum development, curriculum review, program evaluation, and problem 
solving. Negotiate, and help various partners negotiate their differences. 
Earn and maintain everyone's trust, and assist all partners to build or maintain 
trusting relationships 

• Evaluate or assist in the evaluation of the program 
Qualifications for all teachers: 

• B.A. (or graduate degree) in education, psychology or social work. (Teaching 
degree) 

• Adult basic education (including adult ESL) teaching background and 
orientation toward adult (not K-12) education. Experience working with "non- 
traditionar adult students 

• Curriculum development experience 

• Counseling skills 

• Able to relate well to adults, caring about adult students, having respect for 
them (so, as one student put it, students can ask questions and "not feel 
dumb") 

• Good interpersonal skills 

• Professional appearance and behavior 
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• Being a learn player" (not too independent, being able to put oneself in the 
shoes of a supervisor, employee, funder, or union representative so that the 
needs of students, management, and the funding source are met) 

• Being patient, fully understanding that progress is usually slow 

• Able to help students at different levels 

• Having a sense of humor 

• Being flexible, not rigid, able to go with the flow 

• Being able to juggle a lot of responsibilities at once 

Desirable Qualities, accomplishments, or experience for all teachers: 

• Peace Corps or VISTA volunteer background, or other significant cross- 
cultural experience 

• Experience working as a worker 

• Understanding the kind of work student/workers do, understanding the 
situation of hourly production or service workers, the nature of their jobs 

• Background in pre-vocational education 
Additional Qualifications for ESL Teachers: 

• Understand, firsthand, through having struggled to learn another language, 
what a student/worker acquiring English is going through learning to think in 
another language 

• Background in English and/or ESL teaching 
Desirable background for ESL Teachers: 
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• Bilingual or multi-lingual (in English and students' first language) 

The above job description is ideal, and not every workplace education teacher 
will fit it fully, it points the way, however, in hiring and in planning staff 
development; if widely shared with workplace education teachers, and with 
those seeking such employment, it may also give teachers a standard against 
which to compare themselves for their professional development. 

IV. Innovations and Strengths at Each Program 

Innovations: 

We asked those we interviewed at all the sites to identify what they thought 
were their program's innovations. Their comments appear below followed by 
our observations on these and other innovations at the programs. 

Kennedy Die Castings 

The evaluation team suggested that Job-specific English language learning 
("shop vernacular") has allowed immigrant employees with limited English 
proficiency to take higher jobs, that this opens doors for people in the shop. 
They also thought that the kind of classroom learning they offer is relevant, 
integrated with learning needed for work on the shop floor, and that this is an 
efficient way to team. 

Kathy Rentsch and Kathy Soderstrom, the education program staff, thought that 
the Worcester Workplace Education Collaborative new adult education 
orientation for the staff of all their programs was an innovation, and that their 
holistic curriculum, which is not driven by isolated skills, and drill and practice, is 
an innovation. With it, students learn real language in the context of actual work 
and life tasks. 

Boston City Hospital 

The evaluation team cited four innovations: 
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1 . A "Managing your Manager unit as an example of the curriculum being 
tailored to this particular workforce; 

2. Bringing into the classroom to talk about their work people who do different 
jobs in the medical field; 

3. Helping workers figure out their pay stubs; and 

4. Workplace-based research projects: for example, reading charts as a 
research project for the science/study skills class. 

Paul Guldenzoph added two: 

5. Flexibility which allows student/employees to do individualized work on their 
lunch breaks; and 

6. The teacher/counselor/coordinator job which allows for a wide variety of 
roles, and (because Paul does everything) fewer meetings! 

Hampden Paper Company 

The education staff at Hampden Paper cited four innovations: 

1. Individualized curriculum development, e.g. the Chemical Mixer Formula 
Ticket; 

2. Use of computers; 

3. Eliciting goals from learners; and 

4. A new "Skills Checklist" which both serves as a menu from which to choose 
learning objectives, and since it was generated from needs identified by 
learners and supervisors, is a record of these needs and issues. 
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South Cova Manor Nursing Home 



johan Uvin suggested that the new theme-based curriculum-as-process may 
allow students at different levels of skills all to engage with the theme, and all to 
improve their language skills. Such themes as: under-employment, mis- 
communication with residents, relationships with English-speaking staff, views 
of learning and teaching, understanding policies and procedures, and career 
advancement, because they affect student/workers at all levels of skills, leave 
room for everyone's participation and language learning. 

Commentary 

The innovations cited at the Kennedy Die Castings, Inc. program may seem 
obvious in the context of workplace education, but against the background of 
traditional elementary, secondary and higher education, they point out that 
workplace education, as practiced at Kennedy Die Castings, and in many other 
companies, is itself an important innovation and that curriculum which 
integrates work and learning is at the center of this innovation. 

We would agree that contextual curricula, particularly at Kennedy Die Castings 
and South Cove Manor Nursing Home, are innovative, and that the processes 
by which these are developed, although in many ways different, are exciting 
contributions to the art of workplace education curriculum development. 

Strengths: 

Each of the four programs has considerable strengths in the area of curriculum. 
From our observations, reviews of materials, and interviews we would bring 
attention especially to the following effective curriculum and instruction features 
of these four programs: 

Kennedy Dte Castings. Inc. 

1 . English That Works, developed by Passeri and Soderstrom, 7/90. This is an 
impressive, home-grown, contextual curriculum developed for Kennedy Die 
Castings, Inc. It is extremely useful to workers and to the company. It also 
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includes, in the Teacher's Reference Guide section, a great list of 
instructional strategies to be used in the classroom; 

2. Closeness to the context of students' lives and needs, to real workplace tasks 
and to other important tasks, problems, issues, and concerns of 
student/workers' lives; 

3. Theme-based instruction which addresses the needs of a multi-level 
classroom, and which includes integrated reading and writing as well as 
listening and speaking; and 

4. Exceptionally talented teaching. We observed in the classroom Kathy 
Soderstrom's use of a very wide variety of methods and materials, the 
integration of listening, speaking, reading and writing skills in theme-based 
lessons, attention to individual as well as group needs, and an exciting pace 
of energetic language learning. 

Boston Cttv Hospital 

1 . The Pre-technica! science and study skills course is an important step in the 
healthcare education and training career ladder, one which is missing from 
most hospitals; 

2. The combination of classes, significant tutoring time, and individual career 
counseling is a strong design. The attempt to level the classes is also 
moving in a positive direction. 

Hampden Paper 

1. Individually-tailored curriculum development; 

2. An individually-paced GED curriculum which appears to be effective in 
meeting students' needs; 



ERIC 



125 



3. Student/workers learning to use computers as tools for word processing, 
accounting, and filing data; 

South Cove Manor Nursing Home 

1. An extremely well-thought-out, theme-based, curriculum development 
process; 

2. Accessibility of classes, given that there is no release time; 

3. Exceptionally talented teaching. The use of role-playing/simulation in Johan 
Uvin's class, the dramatic enactments of high-interest problems to be 
examined, and solved with newly acquired knowledge and language, is 
intensely involving and motivating. 

4. An exciting collection of high-interest, authentic, home-grown, site-specific 
teaching materials which are used in enacting scenarios of problems drawn 
from the contexts of students' experiences. 

There are also some strengths across the four programs which we 
observed. 

1. Every program addressed student/workers' basic skills needs in the contexts 
of work and other daily living needs, as perceived by management, labor, 
students themselves, and instructors. This is facilitated by the state and local 
partnership model. 

2. Every program employed talented teachers and teacher/coordinators who 
had experience teaching adults, and who believed in a learner-centered 
approach (although these were implemented differently at each program). 

3. Every program was engaged in some level of original curriculum 
development or adaptation of curriculum. In every program, curriculum was 
dynamic, changing to better respond to the needs of student/workers and the 
workplace. 
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4. At least three of the four programs had published student writing: for 
example, Kennedy Die Castings' student biography books, Hampden Paper 1 
"Our Words", which includes student writing and photographs, and the South 
Cove Manor workplace education program newsletter and student writings 
collections, "Write From the Floor." 

5. All four programs make a number of efforts, in person, and through written 
materials, to keep their programs visible within the workplace settings, and 
to both draw upon and influence "company culture". 

Other Observations: 

As we reviewed the findings, the many references to "culture* "cultural 
sensitivity" and "cross-cultural knowledge and attitudes" reminded us that 
across the Workplace Education Initiative, the cultural context is critical.. 
Although this was not an area which this study intended to address specifically, 
it was clear that it is one which programs must consider. Needs for workers to 
learn about and be sensitive to each others' cultures; for supervisors to learn 
about workers* cultures; for newcomers to learn about "American" culture; and 
for everyone to understand and have a part in shaping "corporate culture" were 
referred to throughout our interviews. 

These needs can and are being addressed in a number of ways. Programs 
suggested hiring teachers who have experienced the struggle of learning 
another language, and understanding another culture, people who are bi- or 
multi-lingual and bi- or multicultural. The curriculum and teaching practices and 
materials can draw upon cultural comparisons in the context of instruction. 
Teachers can help students to understand the similarities and differences 
between the company/corporate/institutional culture, majority culture in the U.S., 
and their own ethnic cultures on such critical work-related issues as: goal 
orientation, the use of time, family responsibilities, the use of one's first 
language, the use of formal and informal English, giving and receiving praise 
and criticism, etc. 
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V. Areas for Growth at each Program 

These are all relatively new, and small programs; compared with larger 
programs with more teachers, space, materials, and other resources, there are 
many areas in which there is room for their growth. 

There are several problems common to all programs: 

1. Students who enroll in these programs will have difficulty moving from the 
most beginning levels up through a high school level, if this is their goal. If 
they cannot move to the next level and receive adequate services, the gains 
they make in these programs may be lost. 

The programs which do not offer a full range of basic education services, 
from basic literacy through r,ost-secondary preparation, are not well 
articulated with other programs in the adult education system which offer the 
education services they lack. In some cases, where there is an articulation 
plan, because of lack of funding in the system, students are not able to get 
seats when they are ready to move to the next program. For example, the 
Kennedy Die Castings program is part of a larger community college 
collaborative, which in the past has been able to provide students who were 
ready with seats in programs at the next levels. This year, seats were not 
available. 

Programs such as those at Boston City Hospital and Hampden Paper 
Company which do offer a wide range of levels have spread their resources 
too thin to deliver all these services equally well. In both these programs, 
one person is asked to teach everything: ESL, basic literacy, five high school 
level GED subject areas, and computer literacy. Even the most gifted 
teacher must find some of these tasks a challenge to do well, and if that 
teacher is replaced, finding someone who can handle the same areas 
equally well is also a difficult challenge. 

2. None of the four programs we studied has a systematic annual review of the 
curriculum by the advisory boards. In our opinion, curriculum review, and 
review of program outcomes, should be major advisory committee activities. 
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3. There are no computers at three of the four programs, and computers are not 
yet well used at Hampden Paper. While they cannot replace the need for 
teachers, they can greatly assist the teacher who knows how to use them 
well. With training, and good software, such a teacher should be able to rely 
upon computer-assisted instruction for math teaching, for example. And 
some computers now have digital sound applications which would be very 
helpful for ESL students who wish to practice on their own. And word 
processing can be a very positive and powerful tool for encouraging writing. 

There are also specific areas for growth at each of the four programs. 

Kennedy Die Castings. Inc. 

1. The evaluation team members raised a concern that four hours per week of 
instruction isnt enough time. We concur. 

2. Despite the success which this program is having with curriculum writing, 
more time is needed to create and critique materials, especially for 
instructors. 

3. We were told that although students are not formally tested using 
standardized tests, students get periodic, informal feedback on their learning 
progress from dialog journals, oral and written quizzes. Further work may be 
needed to link these informal assessments with the curriculum so that 
students are always clear about what they have achieved and where they 
need to improve. 

4. Students are not involved enough in curriculum decision-making or program 
development. 

Boston Citv Hospital 

1 . The evaluation team cited the need to get more materials and tasks from work 
integrated into the classroom. 
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2. They also cited the need for student follow-up six months after each class 
has ended to make sure that student/employees are firmly on the next step 
of the career ladder. 

3. The pre-tech science and study skills curriculum, in our opinion, is adequate 
but might be improved by using more of the strategies and techniques 
included in community-college level "Learning to Leam" courses. For 
example, such an approach stresses how to generate good questions from 
lecture notes, rather than listening and outlining. (See Recommendations 
section for further information on this approach.) 

Hampden Paper 

1 . The curriculum needs more focus. In an attempt to meet all student needs, it 
is currently spread too thin. 

2. Measuring and recording learner progress needs further development. This 
problem is, in part, the result of having an unfocused curriculum. For 
example, one part of the curriculum is very focused: QED preparation. The 
Jobs For Youth GED curriculum which the program uses is a complete, 
individually-paced, objectives-based curriculum with a clear method of 
recording learner progress (for student/worker and teacher). The evolving 
"skills List" may, over time, become focused in this way, but will require 
significant more time, and possibly some curriculum development expertise, 
for this to be achieved 

South Cove Manor Nursing Home 

1 . The Administrator at South Cove Manor expressed a concern that the 
curriculum should include more job-related reading and writing tasks. We 
concur that more reading and writing can and should be integrated in 
classroom activities. The Kennedy Die Castings Program provides one 
model of how this can be done with beginning level ESL students. 
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2. One area of growth is assessment on learner progress. This is a particularly 
difficult task in a theme-based curriculum, but Johan Uvin believes that the 
outcome study has helped South Cove Manor move ahead in this area. 

3. Johan Uvin identified staff development as an area for growth. 



VI. Recommendations 

Recommendations for Each Program: 
Kennedy Die Castings. Inc. 

1 . We recommend increasing the number of hours per week per class to meet 
the minimum optimal standard established by the Department of Education 
for Beginner ESL programs, 6 hours. 

2. We recommend that the Worcester Workplace Education Collaborative 
explore the possibility of offering advanced ESU ABE, and adult secondary 
education classes with guaranteed seats for student/workers who are ready 
for more advanced levels. These might be offered within the consortium at 
various companies, or might be offered at the College. Perhaps companies 
would be willing to purchase a minimum number of seats each year, and 
additional seats on an as needed basis. Although a disadvantage of such a 
model would undoubtedly be a loss of site-specific curriculum, it would be 
outweighed by the advantage of allowing students to continue in their 
studies beyond what the company is able to afford on site. 

Boston City Hospital 

1 . Boston City Hospital has several, separately-funded workplace education 
classes. Although these are under one hospital department, they are 
delivered through both outside education providers and hospital employees. 
There may be an opportunity to bring these together programmatically and 
to develop a healthcare basic education system within the Boston 
Department of Health and Hospitals which offers opportunities from the most 
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basic level up through post-secondary level. If so, we would recommend 
offering more leveled classes, in particular, breaking up the ABE class. 

2. Boston City Hospital could benefit from examining the thinking and study 
skills instruction courses at community colleges and in some workplace 
education programs, for inclusion of this approach in the pre-tech science 
and study skills course. (See Appendix C for references.) 

3. The advisory committee and evaluation team at this program need to have 
more input into curriculum development. 

Hamprton Paper Company 

1. The education program staff and advisory group should agree upon a 
definition of curriculum, and focus upon two or three well-defined areas for 
curriculum development in the next year or two. Three possible areas upon 
which to focus might be: 

a Job Certificate Preparation Curricula 

As we understand it, Hampden Paper is moving toward a system where 
workers will be able to demonstrate competency in a job area, and 
receive a certificate of competency. An employee would be encouraged 
to acquire as many of these as possible, and there would be financial 
incentives for doing so. For many employees to succeed in mastering 
these competencies, they will need basic skills as well as job-specific 
skills. Indeed, the two will be interwoven, because listening, speaking, 
reading, writing, computing, analytical and synthetic thinking, problem 
solving, and teamwork skills will, no doubt, be required to attain many of 
these competencies. 

There is an excellent opportunity, as the certificate competencies are 
developed, to do a task analysis on the basic skills required for each 
certificate, and to develop for each an Entry Standards Assessment 
(ESA) for each certificate area. The ESA would help an employee know if 
he already had the basic skills needed for that certificate area, or exactly 
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what basic skills he needed to improve. Then, curriculum materials could 
be developed which helped student/workers team the basic skills in the 
context of work-related problems. Another form of the ESA could be used 
when the student/worker was ready to demonstrate basic skills 
competencies and move toward acquiring the rest of the certificate 
competencies. 

The Chemical Mixing Formula Ticket is an example of this job-related 
basic skills teaching. It could be expanded so that there were 
instructional materials like this tailor made for every certificate area 
where student/workers lacked needed basic skills 

There is a body of knowledge for accomplishing this kind of curriculum 
development which Hampden Paper might draw upon, for example, the 
Dacum process, or the San Mateo County, California ACE program. 

b. Computer-assisted instruction 

As the company becomes clearer about what tasks workers will need to 
perform with computers, these should be shared with the workplace 
education program staff and curriculum could be developed to help 
workers learn and/or practice these ^ks in the learning center. 
Computer-assisted instruction software could be purchased which helps 
student/workers to: prepare for GED subjects; acquire basic math skills 
(there are many reasonable good, objectives-based or skills-based basic 
math software programs); and learn keyboard skills (there are many 
excellent software programs for this). Computer-assisted instruction may 
also be useful in other areas. 

We also recommend purchasing additional computers for the learning 
center, and linking them in a Local Area Network, with shared printers. A 
laser disc player might also be an investment worth considering, and if 
there will be significant curriculum development activity, we believe a 
high quality scanner might prove useful. 
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c. Fully develop the "Skills List" into an individually-paced basic skills 
curriculum for Hampden Paper. Starting with the existing list, expand the 
needs assessment process so that more competency areas are defined 
(by supervisors, student/workers, union representatives). Then do a task 
analysis for each of these so there is a clear understanding of what basic 
skills tasks are required for each area Relate these to a larger taxonomy 
of basic skills objectives, and develop performance-based measures 
and/or test Hems for each objective. When fully developed, the skills list 
should be a list of Hampden Paper task areas of interest to workers and 
supervisors, with specific basic skills objectives and measures for each. 

2. Students should be encouraged to attend the learning center for a minimum 
of 6 hours per week, the minimum optimum standard the Department of 
Education has established for ABE programs. 

3. A private space should be made available as close to the learning center as 
possible for counseling individual students. 

South Cove Manor Nursing Home 

1 . Although it is not always possible to find a teacher who speaks the native 
languages of the students, this is an important consideration if new teachers 
are hired. 

2. The administration at South Cove Manor draws a sharp distinction between 
work-related language skills and li'9 skills, and insists that work-related 
language skills must be the exclusive emphasis of this program. Although 
one can draw a distinction, the categories blur, and some "life skills" may be 
more important to worker productivity that "work-related" language skills. 
Workers who cannot solve critical daily living problems because they do not 
have the English language skills or cultural/social knowledge will be 
affected by their problems, at work as well as in other parts of their lives. For 
this reason, we recommend that as student/workers identify critical language 
needs-whether at work or in other parts f their lives-these be addressed in 
the curriculum and in class. 
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3. As of this writing, the continuation of funding for this program is uncertain. 
We recommend that every opportunity be made to find the resources to 
continue it. It is a model program, with an outstanding approach to 
curriculum, which in our opinion, should not only be continued, but emulated 
in many other workplace education programs. 



Recommendations for the Initiative: 

The Massachusetts Workplace Education Initiative has wisely encouraged each 
of the programs to develop its own curriculum, tailored to meet its own local 
needs. However, at this point in the development of the Initiative we believe 
that the programs could benefit from a more systemic approach. We 
recommend the following actions and indicate in parentheses who might take 
responsibility for them: 

1 . Continue the evaluation team support to current, as well as new programs, to 
enable programs to define and measure program outcomes and to evaluate 
programs' diagnostic/placement and student progress assessment systems 
(Evaluation Team); 

2. Provide an opportunity for programs to share, publish, and compare their 
intended outcomes (State Level Partners); 

3. Ask advisory committees and those responsible for curriculum development 
at each site to review their curriculum and classroom assessment 
instruments in terms of these intended outcomes, and to refine (add and 
eliminate) curriculum units and instructional activities to increase 
enhancement of these agreed-upon goals (Program Sites); 

4. Examine intended outcomes across the initiative to determine which of these 
goals are widely shared, and publish a document listing these. Ask 
individual programs to list commercial and home-grown curriculum materials 
which they have found effective in helping student/workers achieve these 
outcomes; and prepare, publish and distribute a matrix of these materials 
indexed to their intended outcomes. (State Level Partners); 
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This will serve three purposes: 1) it will provide a useful self-evaluation 
activity for each program as it sees for which goals it has or doesnl have 
adequate instructional materials; 2) it will provide a useful way for programs 
to share materials across the Initiative; and 3) it will provide a clear picture of 
where materials need to be developed across the initiative (State Level 
Partners, through SABES); 

5. Using the set of draft curriculum standards which we prepared for this study, 
or another draft set of curriculum standards, ask representatives from all 
programs in the Initiative to review, critique, and accept a set of initiative- 
wide curriculum standards. Use these standards in requests for proposals, 
and use them for periodic peer evaluation of workplace education programs 
(State Level Partners and Program Sites); 

6. Staff development is important for ell workplace education teachers and 
other staff. Each staff member should have an annual staff development 
plan which is related to areas of the job in which this person agrees that 
further knowledge, skills, or practice are needed. (State Level Partners and 
Program Sites) 

7. As workplace education teachers need to share their problems, solutions 
and expertise, we recommend that regional workplace education teacher- 
sharing networks be created in at least two other parts of the state, the 
central and western regions. They could be modelled after the successful 
existing network in the Boston area sponsored by the ALRI/SABES Greater 
Boston Regional Center. These networks could use the above matrix as a 
tool for teachers sharing curriculum mateials and for collaboration to 
develop new materials across programs (State Level Partners, through 
SABES); 

8. Copies of workplace education curricula and curriculum materials developed 
locally and elsewhere should continue to be collected at the ALRI/Greater 
Boston SABES Regional Center library and at the State Workplace 
Education Library, and lists of these materials should continue to be updated 
and circulated among workplace education programs (SABES); 
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9. The Orientation for adult basic education staff developed by Quinsigamond 
Community College for its adult education teachers should be reviewed by 
SABES as a possible model for orienting new workplace education teachers 
regionally across the state (SABES); 

10. The position of state Workplace Education Coordinator should be restored 
in order to coordinate these and other state-level responsibilities. (State 
Level Partners). 

1 1 . The workplace education programs should consider the advantages of 
computer-assisted instruction, and purchase computers. The state staff 
development and support system should provide training to workplace 
education teachers in how to use computers for instructional purposes and 
as tools for word processing, and information storage and retrieval. (State 
Level Partners, Program Sites, and SABES) 

12. As in many adult basic education programs, the systematic teaching of 
mathematics is weak or non-existent in these programs. Systematic 
mathematics instruction, possibly using computer-assisted instructional 
software, should be made a higher priority, particularly in programs where 
mathematics knowledge and skills are required to perform work tasks. 
(Program Sites) 

13. If the use of computers in workplace education programs in small to 
medium-sized companies is to become a reality, resources for computers 
and computer-assisted instruction need to be made available to programs 
through both public and private sources. In particular, we recommend that 
the Federal Workplace Education Initiative address this problem in its 
guidelines an requests for proposals. 

VII. Conclusions 

From a limited, not necessarily representative sample of programs we must be 
cautious in drawing conclusions about the curricula found across the Workplace 
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Initiative as a whole; yet some patterns or trends are apparent to us even in this 
small sample: 



1. Curriculum is holistic, participatory, created or tailored at each site, and 
addresses needs of student/workers, companies, and organized labor. 

The Massachusetts Workplace Education Initiative definition of curriculum, 
"the materials developed and used by program staff, the process by which 
materials are put together, the actual classroom practices, and the 
philosophy which guides these activities" leaves room for great diversity of 
curricula at the program sites. At most of the sites we observed, there was a 
clearly articulated curriculum philosophy, which usually included an 
emphasis on creating or tailoring curricula (i.e. instructional practices, 
materials created and adapted) to individual student/workers' needs and 
goals, which were described in individual learning plans for each 
student/worker. The site curricula were also company-centered and, where 
there was a union partner, union-centered; needs and goals of management 
and organized labor were specifically articulated, conscientiously 
addressed, and evaluated in the programs. 

2. The role of teacher is critical, and requires experience and talent. 

The success of all these programs depends on having teachers who have a 
wide range of basic skills teaching experience and knowledge (basic 
reading, writing, math; ESL, secondary level subject areas, computer 
literacy) as well as a background in counseling, curriculum development, 
and possibly administration and program evaluation. Such a teacher must 
also have an understanding of adult education principles, be able to relate 
well to adult learners, and work effectively in a workplace environment. The 
workplace education teacher job description which was created from data 
gathered in this study shows in detail that this is a tall order. 

Although most of the teachers we observed met most of these expectations, 
some to an extraordinary extent, nevertheless, continued staff development 
must be a high priority for all these programs. 
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3. Instructional methods and materials are varied, but computer-assisted 
instruction Is lacking. 

None of the four programs rely on a "workbook approach," although several 
use workbooks as one source of instruction. Teacher modelling of skills, 
role-plays, tutoring, case studies, drill and practice, learning vocabulary from 
on-site responsibilities, viewing videotapes, co-operative learning activities, 
dialog journals, choral work, games, language experience charts, and 
discussing job-related tools and parts are a few of the many methods used. 
One area, however, which needs further attention is the use of computers: 
computer-assisted instruction (including computer-assisted language 
learning), and the use of computer tools such as word processing. Only one 
of the four programs even has computers available for student use. 

4. Student/workers participate in curriculum decision-making. 

To an unusual degree, student/workers participate in decision-making about 
these programs, and about the curriculum. In two programs, either as a 
group or as individuals students have a systematic say in the curriculum they 
pursue. Even in programs where this is not systematically reinforced, 
students feel they have a significant voice which is respected by the program 
partners. 

5. The curricula are work-related - and more. 

Not surprisingly, the focus of the curriculum, in all four programs, is work- 
related basic skills. Yet, there is a surprising and welcome integration of 
other skills and knowledge which student/workers need both at work and in 
the rest of their lives. No doubt this is because of the participatory nature of 
several of these programs which is encouraged by the state partners. 

6. The programs' curricula are meeting the Initiative standards. 

At the time of this study, curriculum development-and the programs 
themsetves-were still relatively new, and none of the programs had what 
might be described as a mature, fully developed, well-tested curriculum. 
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Yet, taken as a whole, the four programs were very well aligned with the 
norms established in the major documents which represented the 
Workplace Education Initiative standards. (These standards may be found in 
the document, "Workplace Education Curriculum Standards" prepared for 
this study and included in the appendix.) 

7. An unsolved problem is how to meet the full range of student/workers' basic 
skills needs with very small programs. 

The most troublesome problem of the design of these programs is that 
because they are small, and not well articulated with other basic education 
programs, a student/worker who completes the available instruction offered 
at the workplace and who wishes to go on to the next level, may not be able 
to do so. Two programs attempt to address this by providing a whole 
continuum of sen/ices from basic literacy and ESL up through secondary 
level, delivered by one teacher - a one-room schoolhouse approach. Two 
programs attempt to address this by referrals out, but because nearly all of 
the programs in the adult basic education system have long waiting lists, this 
is not always effective. 

8. The curricula have an impact on workplace culture. 

There is some evidence that the curricula of these programs have an impact 
on workplace culture, that they are an important part of an organization-wide 
effort to move toward "high performance organizational models" which 
require worker participation in quality control decisions, and which require 
higher basic skills. 

9. Site curriculum products and processes could be refined and disseminated. 

There are some very creative curriculum materials development efforts 
under way. If encouraged and supported, they could result in well- 
developed processes and products for dissemination. Most notable are 
those at Kennedy Die Castings, Inc. and at the South Cove Manor Nursing 
Home. 



ERIC 



140 lull 



Termination Study 



Introduction 

The termination study was not part of the original plan for the MWEi Cycle 4 
evaluation, it was bom midway through the evaluation year when one company 
- Friction Materials, Inc. (FMI) - ceased operation and another, Spir-it, Inc. 
(Spir-it) became unable to sustain its education program. Both of these 
companies had been offering education services to their workers for over two 
years and had participated in former evaluations. Their established 
membership in the MWEI, the amount of information already collected on them 
and their unanticipated terminations made these programs worthy of continued 
study. The state steering committee and the evaluators agreed that there was 
much to be learned from answers to why these these programs ended. 

The termination study forms the third section of this report. It: 

(1) describes and analyses how the goals of the FMI ESL program were linked 
to company growth and profitability and examines why the program- and 
the company- could not fulfill its goals. 

(2) examines why Spir-it, Inc. was no longer able to sustain its education 
program, and addresses in particular what role basic skills education played 
in the company's long-term strategy for bringing about needed 
organizational change. 

Methodology 

The evaluator conducted a series of interviews with program staff and company 
representatives at each site, in each interview, the evaluator followed a 
protocol which was designed to help respondents: 

(1 ) reflect on the original goals of their program; 

(2) consider the extent to which those goals had been met; 

(3) think critically about why those goals were met or not; 

(4) think creatively about how their program could have been structured to avoid 
the problems it encountered. 
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In addition, respondents were asked to answer a series of questions about 
program outcomes. These questions are similar to those used by the Hampden 
Paper and Aerovox evaluation teams in their outcome studies. These questions 
use the American Society for Training and Development "essential skills 
categories" as a framework for assessing the impact of the workplace education 
programs at FMI and Spir-rt. (See Appendix D, under separate cover, for copies 
of the protocols used.) 

Interviews lasted from a half hour to three hours, depending or how involved 
the respondent was in the program and how much time s/he had available. At 
FMI respondents included the program coordinator, the business agent for 
Local 2542 of the Amalgamated Clothing and Textile Workers Union (ACTWU), 
a student in the program, a supervisor, the statistical process control specialist, 
the director of personnel, and the plant manager. 

At Spir-it, respondents included the program coordinator, a student in the 
program, a supervisor, and the director of personnel. The evaluator had 
already conducted the first meeting with the Spir-it evaluation team as part of 
the Spir-it outcome study. After it became clear that Spir-it would not be serving 
as an outcome study site, the evaluator used the information collected in that 
meeting to inform the termination study. In addition, the evaluator attended two 
meetings with the MWEI Coordinator and Employment Connections, Inc. (EC1) 
administrative staff. ECI is the learning provider agency for the Spir-it program. 
One of these two meeting was also attended by Spir-it's director of personnel. 

Friction Materials, Inc., Lawrence 

Background 

FMI is a small (100 to 300 employees), fifty-year-old company formerly known 
as the P.T. Brake Lining Company, a trademark name in brake technology. The 
company was founded and run by a man who knew every part of every machine 
and every worker by name. When his sons inherited the business, they faced 
two problems: they did not have the skills to run the company in the same 
hands-on way their father had, and the brake technology industry was 
changing. Asbestos had been a main component of brake technologies since 
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the start of the automotive industry and federal regulations were beginning to 
restrict the use of this toxic substance. Five years ago, the company was on the 
brink of bankruptcy, in part due to heavy investments in new, non-asbestos- 
based materials, and losing customers from a bad reputation for poor quality 
control. As a remedy, the once-successful P.T. Brake Lining Company 
negotiated a transfer of assets to Eckfln, Inc. Ecklin owns four-to-five dozen 
companies like FMI throughout the U.S. and Canada, many of which 
manufacture automotive parts. Ecklin was founded in the 1920s and has grown 
over the years to a $1.5 billion a year operation with 12,000 employees. Ecklin, 
Inc. changed the name of the old company to Friction Materials, Inc. There are 
five companies in the Ecklin group which manufacture brake products and FMI 
is one of them. 

Workers who were still employed by the company prior to the transfer became 
concerned about how the transfer might affect them. They voted in a new union 
(ACTWU Local 2542) by a narrow margin (53 to 46) in February of 1987 . 
Some observers hold the opinion that Ecklin frowns on having its companies 
unionized. These observers suggest that FMI's unionization might have caused 
the parent company to judge FMI harshly when it failed to show a profit quarter 
after quarter. But, in the beginning, the newly-unionized FMI stepped up 
production with the backing of the parent company, with the hope of reclaiming 
some of its former market share. By February 1989, when the FMI on-site ESL 
program was begun, FMI had grown from 100 to 260 employees. 

The Program 

The ESL program, which was developed in partnership with the Lawrence Adult 
Learning Center, hoped to address some of the problems the company still 
faced despite its growth: (1) poor retention of workers; (2) poor communication 
between English-speaking supervisors and Spanish-speaking workers (95% of 
the FMI workforce speaks Spanish as a first language); (3) lack of ability of 
many workers to participate in a company-wide program to introduce statistical 
process control- a program which was essential to wooing back lost customers 
and opening up new markets; and (4) financial loss from excessive waste 
(scrap) in the production process, also to be addressed by the new SPC 
system. 
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Classes met twice a week for two hours- a total of four hours a week. For a 
short time, release time was granted to workers who attended classes. But 
soon workers were asked to attend class after the main shift , with pay for two of 
the hours they spent in class. With the help of a supervisor who saw the ESL 
program as a critical component of the company's effort to boost production as 
well as morale, the teacher/coordinator developed a curriculum that was work- 
specific. The results of worker participation in the ESL program were quick and 
dramatic. They included: 

Workers improved their mastery of important job-specific vocabulary and, 
therefore, of certain job procedures. 

- After two weeks of instruction, the company saw a 40% reduction in 
scrap. (The scrap rate had previously been about 60%; in October 
1989 it was 16%. Before the ESL class, the company was throwing 
away more in scrap than it was using in products.) 

~ Production improvenents of about 20% on one line manifested in the 
first week, thereby increasing salary for workers. 

- The personnel office documented improvement in the company's safety 
record. 

~ Retention of workers increased. 

In late winter/early spring 1989, with enrollment around 60 (ten above the 50 
workers which the grant targeted), it seemed that the FMI program was one of 
the best examples of a successful workplace education program in the 
federally-funded MWEI cohort. Then the lay-offs hit. By late summer, over 100 
workers had been laid off. Classes were reduced by over two-thirds, leaving 
only 14 workers enrolled. Workers with no seniority were the first to go, and 
they were the majority of those enrolled. 

The reason given for the lay-offs was lack of orders caused by a small 
recession. Ordinarily, distributors and large transit companies keep their 
shelves well stocked with brake linings. In a recession, their business slows 
down; they don't buy as much; they take the risk that they can buy quickly as the 
need arises. (When a truck company is working full-tilt, it needs to have brake 
linings on i.und. When business slows and trucks are idle, orders dwindle.) 
The lay-offs were probably not as much a surprise to management as they were 
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to some workers. The P.T. Brake Company had lost many customers because 
of its "ship anything" attitude in its later years and the new FMI had not wooed 
these customers back quickly enough. Instituting SPC and establishing a 
workplace ESL program were part of an overall strategy to revive the company, 
but there wasnl enough time, it seems, to go far enough with the plan before 
recession and lay-offs hit. 

Problems 

With hindsight it seems that the company never really recovered from the blow 
which the 1989 recession and first round of lay-offs dealt From the summer of 
1989 until the spring of 1991 , the company suffered several waves of call-backs 
and lay-offs. While it isn't atypical in this industry to experience cycles of fast 
and slow sales (slow sales occurring especially in the fall), the combination of 
the natural sales cycle, a real recession, and lagging orders due to a weakened 
company reputation hit FMI very hard. During this time, much effort went into 
keeping the ESL program alive. Clear communication between workers and 
supervisors was as important as ever. This was especially true since (1) Ecklin 
oegan production of a new, more-complicated brake at FMt~ thelnteg rally 
molded* (IM) line- which required workers to take instructions from supervisors; 
and (2) a company-wide SPC program was being instituted that demanded 
fairly sophisticated English language skills. The ESL program was still seen as 
intimately linked to company g'bals of improved communication in English and 
establishing quality awareness- and seen, by some, as essential to achieving 
those goals. 

With the first round of lay-offs, the workplace education partnership had to re- 
think how to reach workers with ESL needs who were not enrolled in the 
regular classes and who apparently would not enroll- the "hard core" 
employees who had been with the company for a while. The partnership 
decided to add a ten-minute ESL/quality-control component to the six half-hour 
SPC training units which many employees were required to take, while 
continuing regular ESL on a smaller scale. 

The ESL add-on strategy -- or "esoteric English" as it was called- did not work. 
It provided too little instruction to too small a group of workers. While regular 
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ESL classes continued on a greatly diminished scale, the partnership decided 
to prepare a quality awareness curriculum that would be offered to all workers 
for a total of six hours during regular shift time. The curriculum would introduce 
basic concepts of quality control in a language-sensitive way, thereby 
promoting quality awareness and ESL simultaneously. Winning release time 
for the quality awareness program was a big step for the ESL staff. Although 
the company paid for two of the four hours workers spent in the ESL program, it 
was adamant about not being able to give release time during regular shift 
hours. Managers argued that they needed workers on shift to meet production 
quotas- especially now that they were relying on a much smaller number of 
workers. Workers said they could not attend class after work because they 
worked other jobs, had family responsibilities, and had transportation problems. 
Providing quality awareness instruction during shift hours demonstrated 
company commitment to education, and, in particular, to education about 
quality. 

The need for ESL instruction, and its relationship to producing quality products, 
was heightened at this time by increasing pressure to perfect the "IM" line. What 
constituted quality in this line differed considerably from quality in other lines. 
(In other lines, quality is a function of shape and drill patterns. In the IM line, 
quality is a function of color, left and right, and front and rear.) Workers could 
not depend on their knowledge of other products to guide the manufacture of 
the "IM" line, nor could they depend on their understanding of what quality in a 
line means. Supervisors had to spend more time repeating instructions in 
English, and f rustration levels rose. One manager suggested, with hindsight, 
that the company should have trained a few people who were bilingual in "IM" 
production at the other Ecklin plant where the line is made, and then had them 
teach everyone at FMI in Spanish, rather than hobble along trying to 
communicate complex information in English. But everyone felt such intense 
pressure to produce that they missed the proverbial forest for the trees. 

Ecklin announced that it would shut down operations at FMI a week before the 
teacher was to pilot the quality awareness program. Ecklin cited fiscal reasons 
for closing the plant. Ecklin owns five other plants which produce the same 
items as FMI. These plants are utilized at 75%-80% capacity. After four years of 
watching FMI struggle to turn a profit, paying rent in a building they do not own, 
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and paying one of the highest electricity rates in the country, Eckiin decided it 
was best to shut down FMI and increase production in the other plants. 

This report is interested in the relationship between the ESL program and the 
shut-down of the plant. At the time of the Eckiin buy-out, there was great hope 
that FMI would make the organizational changes needed to become a real 
competitor in brake technologies. Managers, supervisors, workers, union 
leaders, and. later, education staff agreed that changes needed to be made. 
And they agreed on what some of the changes should be- improved 
communication in English, improved quality awareness, and instituting SPC. 
The ESL program was one way to bring about those changes. Did the ESL 
program live up to its promise, and was it, then, not enough to help save the 
company? Or was the program never fully realized as part of a genuine 
strategy for change? If it was never fully realized, why wasnt it- given the 
apparent agreement o r . goals among the key players? 

The information collected during the termination study attempts to answer these 
questions. 

Conclusions 

Despite the agreement on goals among the key players, and the remarkable 
outcomes which the program reported in its first three months, the program did 
not live up to its promise. While the program was supported by some managers 
and supervisors who were very aware of how important the program was both 
in the short and long term, there was not enough support from upper 
management to develop it fully or to carry it through hard times. 

* Production against education, not production through education. Under 
pressure to meet quotas with a diminished workforce, management seemed to 
pit immediate production goals against a long-term commitment to education 
and organizational change. This was evident at the most obvious level in the 
no-release-time policy and the minimum hours of instruction offered per week. 
The plant manager himself said: "The atmosphere in Massachusetts is too 
competitive to give time." All the respondents, including the plant manager, 
said that lack of release time was a major barrier to program success. The 
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apparent bind between meeting short-term production goals and making a 
long-term investment In the workforce is not unique to FMI. Increasing 
numbers of reports on education and the economy discuss at length the need 
for American management to shift its focus to the long-term. But this is more 
complicated in an industry where there is no history (and no need until now) of 
understanding the relationship between education and improvement. 

In companies that manufacture electronic or communications products, for 
example, the need to keep up with new ways of producing and organizing 
work is more obvious. The technology itself changes rapidly, so the product 
drives change. The new "IM" line challenged the company's ability to cope 
with new-product manufacture, but there was no infrastructure to help the 
company cope. Furthermore, the parent company, Ecklin, did not move in to 
support education as a strategy when local management didn't. Ecklin's 
reputation, as reported, suggests that it does not understand- and is, 
therefore, not committed to- education as a strategy for business 
improvement. 

• Questionable commitment to quality. Management's commitment to 
instituting a complete quality control program was questionable. The general 
perception of the respondents was that not all managers really bought into the 
quality program, especially important upper-level managers. There was even 
a question abut the extent to which the quality program was intended to be an 
appeasement for one large customer as opposed to a method which 
genuinely guided production and standards for all customers. "Quality comes 
top down, not bottom up " is an axiom of the quality movement. Without 
support from the top, quality could not be integrated into all the levels and sub- 
levels of the production process. 

Some respondents reported that workers who valued quality - and slowed 
their pace to produce it - were told by supervisors that FMI wanted "numbers." 
This happened despite company documentation of the loss of thousands of 
dollars of spoiled goods each day. Ecklin kept many of the former P.T. Brake 
Lining managers and staff when it took over. These personnel continued to 
support the old ways of doing things -- the ways they knew and were 
comfortable with - despite the new demand for quality. Without the full 
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support of the upper-level managers, the staff hired to implement the quality 
program could go only so far. 

Furthermore, the incentive system at FMI stressed piece work over quality by 
paying workers a bonus for pieces finished over a certain quota In this 
system, the message is that, if you produce in quantity, maybe you'll get 
enough good pieces. There is an inherent contradiction in a production 
system which espouses quality production on the one hand and gives 
incentives for numbers of poorly finished pieces on the other. 

if establishing quality throughout all levels of the plant really was a main 
objective of management, teaching English was a limited strategy to pursue. 
Because the plant was almost entirely Spanish-speaking, management had 
the opportunity to provide SPC instruction in Spanish, coupled with an ESL 
program designed to match it This would have reached more workers in less 
time and better demonstrated management's commitment to quality. Workers 
could not take an ESL program that was linked to quality seriously, if the 
quality program itself was not taken seriously. 

• Workers not invested in change . Under these circumstances, the workers 
themselves were not motivated to attend the ESL class. This was in part due 
to the muddled signals being sent by the company about education and 
quality control. Workers also saw little personal benefit in participating in the 
class. Lawrence is a predominantly Spanish speaking city and many workers 
do not perceive a need to speak English in order to live well. With 
unemployment in Lawrence now at 17%, and the need to find work in an 
English- speaking community perhaps more deeply felt, some workers may 
now regret missing the opportunity to team English. But, in the FMI context, if 
the company commitment to quality was questionable, then workers would not 
be motivated to spend four extra hours in the plant learning English, even if 
they were getting paid for two of them. 

The situation is complicated in a company where workers are mostly 
immigrants and speak Spanish, and where managers are Anglos. Deeply 
imbedded attitudes about ethnicity, class, and roles in the workplace may bias 
management's perceptions of what workers can do. Managers believe they 
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give workers a great opportunity to ieam, do not see the limitations built into 
the opportunity, and then blame the workers for not being interested or 
motivated enough to take advantage of it. The ESL program opened the door 
to involving workers in decisions about how to save material, promote safety, 
and keep the workplace clean. It got workers excited about honing their skills 
and doing quality work. But at FMI the door was not opened very far. 

The following example captures some workers* feeling that they were not 
respected as full partners in the FMI make-over: A worker was upset that he 
had to follow instructions from a supervisor who consistently made mistakes 
himself and instructed others poorly. The worker brought "to the table" an idea 
about how to save materials, but "the company didnt listen." Workers 
perceived those few managers who were on their side as blocked by others 
with "more power but less insight." 

Union leaders and union members looked on in dismay. But, for the most 
part, the union/management relationship was good. It is unlikely that the 
presence of a union tipped EcWin's decision to H ose down the plant. 

Could a more comprehensive ESL program, integrated with a quality training 
program, have saved FMI? This evaluation can only raise the question in an 
effort to stimulate thought and action in other companies, education providers, 
and state agencies responsible for making workplace education programs 
successful. The evidence and analysis presented here are intended to guide 
discussion about workplace education in ways that link it to broad company, 
union, and worker goals. 



Splr-it, Inc., Wakefield 
Background 

Spir-it , Inc. is a 35-year-old company which produces plastic stir-sticks, straws, 
picks, spears, and food markers. It is the sole manufacturer of plastic coffee 
stirrers for MacDonald's world-wide, and the production of this one item 
accounts for 20% of Spir-it's entire operation. With annual sales of $8.5 million 
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in the mid-eighties, Spir-'rt cornered 60-65% of the market share, selling to 
clients like Eastern, Pan Am and KLM Royal Dutch Airlines, Hilton Hotels, and 
casinos in Las Vegas. Purchased from the original owner in 1988 by an 
investment group (with economy stabilizing restrictions, such as the firm having 
to remain in Wakefield), the company hoped to grow over the next two years to 
annual sales of $15 million. 

Management's plan to almost double sales in a two-year period, coupled with 
the need to automate production to meet changing customer demands, 
underscored the need for on-site instruction for workers whose basic skills and 
English-language proficiency were limited. For example, MacDonald's would 
soon require ail coffee stirrers to be individually bagged in cellophane. (Until 
recently, only stirrers used outside the United States we> « individually bagged; 
those used domestically were not.) The machines at Spir-it which bagged the 
stirrers for years could not keep up with the additional numbers of stirrers which 
needed to be bagged. Engineers designed new computerized "bagging" 
machines which required operators to use basic math to run them and to 
perform quality control functions on them. 

Supervisors noted that there was a big problem because workers could not 
understand directions given to them in English. This problem affected all levels 
of productivity- from the amount of scrap produced to the amount of down time 
on a machine which could actually be repaired with a little information and 
know-how. Feelings of inadequacy and lack of confidence also affected 
relationships among workers and their approach to work. There was general 
agreement, therefore, among supervisors and upper-level managers about the 
benefit of providing instruction which would help workers to: 

(1) learn enough English to understand and read directions; 

(2) understand their machines well enough to stop production when the 
machines break down; and 

(3) develop the confidence to call a supervisor to ask for help with solving 
problems, and to fix the problems themselves when possible. (For 
example, a product was run for three days in the wrong color before 
someone noticed and did something about the problem.) 
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In the spring of 1989, Employment Connections, Inc. (ECI), located in Chelsea, 
began an ESL program at Spir-it for a group of about eight workers. Enrollees 
were mostly Hispanic; a few were Cambodian or Vietnamese. The employee 
population was (and is) somewhat transient, and management was aware that 
the need for ESL could increase significantly in a short time, (in 1987, 50% of 
the workers were Cambodian.) 

The personnel director estimated that approximately twenty out of 125 
employees could benefit from ESL instruction. With 8 enrolled, more than half 
of the workers in need of instruction were not asking for it. The company had 
learned that forcing participation was not successful, and left it up to individual 
workers to enroll or not. There wa: no paid release time and workers attended 
class on their own time after the main shift. Coordinating transportation and 
daycare were two significant barriers to participation. Program staff did not 
survey employees to see how many were kept from attending class due to 
transportation or daycare issues, but the observation was that "(daycare and 
transportation problems) affect every female in the class." Classes were 80% 
female. 

The classes joined students together at the beginning, intermediate, and 
advanced ESL levels, but numbers were small enough to make the class 
manageable. Some preparation was made to provide math instruction but the 
coorcfinator/teacher left the program abruptly. It then took some time to find a 
replacement and get the program up and running again. 

After its first year, the program seemed successful. The following are some 
outcomes which document that success: 

• While putting together materials for hourly employee i. ,'rfcrmance reviews, 
the director of personnel noted changes in the prod »ciiv» j levels of seven 
of the employees enrolled in the ESL class; productivity levels for these 
employees increased from 2% to 10%. The director of personnel was not 
comfortable attributing the increase solely to improved proficiency in 
English (she suggested that increased attention may also be a factor). 
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Nonetheless, positive results were noted according to a very specific 
company measure. 

• Supervisors observed the following results from ESL instruction: 

- Workers could identify what's wrong with a machine and get a 
mechanic to fix it without a translator; 

Total scrap decreased from as much as 1 5-16 pounds per day to 5- 
6 pounds per day; 

- Workers increased in self confidence and satisfaction with being at 
work; 

- Workers showed greater independence in job performance and 
increased cooperation with co-workers; and 

~ Workers showed increased appreciation for the role of education.in 
the workplace. 

Problems 

The first two 12-week cycles of the second year of classes had reasonable 
enrollments - 8 and 14 students respectively. When the third cycle began in 
January of 1991 with only 3 students, program staff and company management 
saw it was time to reevaluate the purpose and format of the ESL program. 
Transportation and family concerns made it difficult once again for even the 
most regular students to come to class after work. The company had also 
suffered some dramatic changes which would influence its re-evaluation. 

The recession hit Spir-it the way that it hit other small manufacturing companies 
in Massachusetts and the nation. When Eastern Airlines collapsed, Spir-it lost 
one big client. When Pan American slowed it services, Spir-it lost again. Other 
potential clients refused to do business with Spir-it because they did not have a 
quality control program in place. As a result, Spir-it either failed to show a profit 
for all four quarters in the second program year, or showed only a very small 
one. The ten managers and operators operating Spir-it for the Board of 
Directors decided to finally make the changes they had been talking about for 
some time: (1 ) consolidate production functions into fewer departments; (2) 
automate the bagging of MacDonald's stirrers; and (3) institute statistical 
process control. 
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Nonetheless, positive results were noted according to a very specific 
company measure. 

• Supervisors observed the following results from ESL instruction: 

- Workers could identify what's wrong with a machine and get a 
mechanic to fix it without a translator; 

- Total scrap decreased from as much as 1 5-16 pounds per day to 5- 
6 pounds per day; 

- workers increased in self confidence and satisfaction with being at 

work; 

- Workers showed greater independence in job performance and 
increased cooperation with co-workers; and 

- Workers showed increased appreciation for the role of education.in 
the workplace. 

Problems 

The first two 12-week cycles of the second year of classes had reasonable 
enrollments - 8 and 14 students respectively. When the third cycle began in 
January of 1991 with only 3 students, program staff and company management 
saw it was time to reevaluate the purpose and format of the ESL program. 
Transportation and family concerns made it difficult once again for even the 
most regular students to come to class after work. The company had also 
suffered some dramatic changes which would influence Its re-evaluation. 

The recession hit Spir-it the way that it hit other small manufacturing companies 
in Massachusetts and the nation. When Eastern Airlines collapsed, Spir-it lost 
one big client. When Pan American slowed it services, Spir-it lost again. Other 
potential clients refused to do business with Spir-it because they did not have a 
quality control program in place. As a result, Spir-it either failed to show a profit 
for all four quarters in the second program year, or showed only a very small 
one. The ten managers and operators operating Spir-it for the Board of 
Directors decided to finally make the changes they had been talking about for 
some time: (1) consolidate production functions into fewer departments; (2) 
automate the bagging of MacDonald's stirrers; and (3) institute statistical 
process control. 
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IV. 



The January 1990 meeting of the Spir-it Workplace Education Advisory Board 
tried to answer this question: How can the educational services which EC1 
offers be of most help to the company in a time of transition? The answer was 
twofold: (1) continue ESL and offer ABE instruction (especially math) in an open 
classroom for one hour, four days a week, with one-half hour per day of 
potential release time, to be approved by supervisors depending on production 
schedules; and (2) design and offer a pre-SPC course on company time for all 
employees which would raise consciousness about quality and prepare the 
way for SPC. "Alt employees" included supervisors, floor workers, set-up 
people, and operators. 

It seemed that the combined open-classroom and pre-SPC curriculum was an 
answer to Spir-it's problems on several levels. An open classroom would invite 
workers to learn English and math with at least partial release time. More and 
more math would be needed to perform quality control functions, and the open 
classroom was one arena in which workers could learn to do more math. A pre- 
SPC course offered during shift time would educate the workforce about the 
range of changes the company planned to undergo and help workers to invest 
in those changes. It appeared that the company was developing an overall 
educational strategy that would help to move it quickly into a high production, 
quality-oriented organization. 

Using an SPC training manual, the coordinator developed the pre-SPC 
curriculum but, shortly before it was to be piloted, the proposed program 
foundered on the issue of release time for line workers. Management decided it 
could not offer any program to workers during shift time as planned because of 
increasing pressure to meet production quotas. Management was adamant 
about not "stopping production"- which, it claimed, would happen if even one 
person were taken off the line. In addition, almost no supervisors were allowing 
workers to come to class on release time because production demands were 
high. As a result, only one or two workers came to class for a total of an hour or 
a little more a week - hardly a basic skills program. 

Management's solution to the problem of not "stopping production" to 
accommodate line workers in the pre-SPC program was to offer the pre-SPC 
program to supervisors and managers who, as salaried workers, could be 
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asked to put the extra time into training without pay. These people would then, 
in turn, teach the set up people, line workers and operators during their shifts. 
This solution did not meet with the standards set by the state funder for proper 
use of workplace education funds for basic skills education. In addition, the 
state coordinator became concerned that supervisors and other managers were 
reneging on the commitment made by the Advisory Board to provide some 
release time for workers, and that an open classroom model for two workers 
was neither fiscally nor educationally sound. 

The MWE1 state coordinator explored several strategies with Spir-it 
management which might encourage hourly workers to participate in the 
program without release time. His intention was to try to save a program which 
the state had nurtured for two and half years. Aware that transportation and 
family issues might still make it impossible for some workers to stay after their 
shift, in one meeting the state coordinator finally suggested that management 
offer a quid pro quo to workers: join the pre-SPC program or open classroom 
now on your own time, and consider it an investment in your job. That is, 
workers would in effect be investing in a company which is trying to improve 
profits in which workers will share through the possibility of promotion. When 
management made it clear to the state coordinator that no such enticement 
could be made because the people who needed ESL and pre-SPC instruction 
most might not have jobs in two months, it became clear that the program could 
not continue. The target group defined by federal guidelines would probably no 
longer be employed. 

Conclusions 

« Production against, edu cation, not production through education 
As with FMI, although it appeared that Spir-it was developing a company-wide 
educational strategy to increase production in the long run, the company 
succumbed to the production vs.education paradigm. The evaluator is not an 
expert in plastics manufacturing and cannot make a technical judgment about 
how easy or difficult it is to take one or more operators off the line for an hour. 
It might in fact be the case that, in a small plastics manufacturing company, 
taking a few people off the line would severely hinder production. However, it 
is difficult to imagine a production process that could not accommodate the 



ERIC 



155 

17o 



regular short-term absence of a few workers if the absences were planned 
properly in advance - and if the absence in the short term were measured 
against the long-term gains that a better-trained worker would contribute. 

Faced with a similar dilemma in another company, a supervisor who is very 
supportive of workplace education said: "The conflict between production and 
going to class is not as bad as people think. Just make a department 'leaner 
for an hour. It's a question of organization. I've never worked in a place 
where you couldn't pull a person or two or three off the line for an hour. You 
just have to think ahead. Show me one plant where there isn't some 
absenteeism. People work around it." This quotation illustrates the framework 
that would make release time possible. This requires that the supervisors be 
"won over" to the side of providing education on site and on company time. At 
Spir-it, this never happened most likely because the upper-level managers 
were not convinced that education was a priority. If they had been convinced, 
they would presumably have communicated this effectively to the supervisors. 

Unfortunately, the management at Spir-it seems to have felt so desperate in 
the face of continually falling profits that they could not embrace a perspective 
that may have helped them improve production in the long run. The pre-SPC 
curriculum was completed without state support and offered to supervisors 
who, presumably, will teach what they have learned, in the best way they can, 
to the workers in their charge. 



Recommendations of the Termination Study: 

These cases provide evidence of the need for the state to systematically 
prepare and disseminate information to employers about education as a long- 
term strategy for change. Case studies, examples from real companies, 
hypothetical circumstances, and results of reports can be presented by resource 
people, or, more usefully, by employers and union people to audiences of their 
own kind. This puts the burden on the state to identify and organize those 
employers and union representatives best suited for the job. Facilitated 
workshops, conferences, meetings and support groups for employers and 
unions are needed. 
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• The MWEI should encourage employers and unions to think about education as 
a long-term strategy tor improving workers' skills and reorganizing the 
workplace. This encouragement may take the form of a workshop, seminar, or 
panel presentation designed for the business community and organized labor. 

• The MWEI should solicit the public support of business and union leaders who 
are invested in education as a long-term strategy for change in the workplace, 
and involve them in workshops and other presentations on the topic. 

• The MWEI should consider more deeply than it has already how ethnicity and 
class influence program success. It should include these issues in discussions 
and workshops about workplace education. 

• The MWEI should consider first-language instruction in those settings where it is 

an appropriate strategy. 
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Appendix A: Outcome Study Case Studies 



CASE STUDY: 
OUTCOME STUDY AT AEROVOX, INC 



Background: Aero vox, Inc., New Bedford 

Aerovox manufactures capacitors- conduits for electrical charges. Aerovox is a 
medium-sized company (about 700 employees) in a mature industry faced with 
the challenge to automate, down-size, develop a quality control program, and to 
re-skill its workforce. These needs must be addressed if the company hopes to 
compete with the less-expensive, equally-sophisticated products being 
produced abroad. Aerovox is the major manufacturer of capacitors in the U.S. 
since its main competitor, General Electric, moved its plant to Mexico. 

Recently, the company purchased a film plant from AT&T to produce DC (direct 
current) capacitors. Accustomed to manufacturing AC (alternating current) 
capacitors on machinery that is not computerized, the company had to adjust to 
producing their new products on highly-sensitive, computerized machines with 
new materials. In addition to this change, the company is using more metalized 
poiypropaiene than before in ail its manufacturing processes. Metalized 
poiypropaiene is a metal-backed paper that eliminates the need to fuse metal 
and paper together in the capacitor manufacturing process. This material is 
central to the future of competitive capacitor manufacturing. The metalized 
poiypropaiene product fine will be highly automated. For example, one 
machine can replace a whole line of operators. 

The loyal workforce at Aerovox is mostly Portuguese-speaking people who 
know their old jobs on the old machinery very well. Within the framework of the 
old production methods, reading and speaking English were not essential skills. 
The demands of performing new jobs on new machines now highlights the 
need to provide English language instruction as a precursor to training on the 
new machines. Even native-English-speakers are challenged to develop skills 
that they did not need or use in the old production method. As the company 
embraces more-sophisticated production technology, workers' inability to 
communicate and read well in English (especially to read manufacturing 
process instructions) or perform math functions required for statistical process 
control (SPC) becomes more problematic. 

The three components of the Aerovox workplace education program- ESL, 
ABE/GED, and math for SPC- ali directly address the need for improving 
workers basic skills to prepare them for new forms of work. The GED 
component does this less directly than the others but the link exists. Workers 
attend class 50 percent on company time and 50 percent on their own time. 

When management looks to where it wants the company to be six years from 
now, education becomes a big piece of the overall strategy. As the company 
continues to down-size, automate, and develop quality control, it will put 
increased demand on workers to problem solve, to cross train in at least two jobs, 
to work together as teams on one machine, and to become more responsible for 
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catching problems with scrap and faults In the production process. The program 
does not seem to operate yet within a company-wide framework of promoting 
education at all levels in the service of continuous improvement. The current 
program is one step in the development of that strategy. 

Description of the evaluation process: 

The nateup of the ev aluation team: The Aerovox evaluation team was 
composed of the vice president for human resources, the personnel manager of 
the Electrical Products Group, the production manager of the Electrical Products 
Group, the department head of 54 workers who were participants in the Aerovox 
workplace education program, two students, and the teacher/coordinator. The 
teacher/coordinator was employed by the Labor Education Center at the 
University of Massachusetts in Dartmouth. 

The process: The Aerovox team followed the ten-step process described in 
Section II of the report of the MWE1 Cycle 4 Evaluation. All members of the team 
participated actively on the team, offering suggestions consonant with their 
interests and roles in the company. Most of the day-to-day work of designing 
and using the data-gathering instruments, and analyzing and presenting the 
data was left in the hands of the program coordinator 

What was learned about the education program: See Section II, Step 8 Of the 
report of the MWEI Cycle 4 Evaluation for a summary of the Aerovox team's 
findings, and the attached report by the program coordinator. 

What was learned about the evaluation process: The Aerovox team was a 
diverse group of people who were unsure of what they could create together 
when they convened for the first time. Because they had never experienced an 
outside evaluation before, they were also unsure about what real help the 
evaluator might give - and how much of the evaluation would be just a time- 
consuming exercise. Over the course of four team meetings, a sense of 
common purpose developed. This was due, in large part, to the efforts of the 
program coordinator who kept in touch with team members h nween evaluation 
team meetings, and who was very invested in having the evaluation succeed 
both in product and process. As tsie preliminary data showed some real gains 
made, the team developed more interest in what the final data might tell. The 
vice president for human resources was especially interested in student self- 
report data on increased safety, as this was consonant with his figures on 
reduced number of worker's compensation claims. 

When the teem members reported their prei binary findings to the vice 
president for quality control, they found a surprisingly sympathetic ear. Recently 
returned from & visit to another company which had embraced the "high 
productivity" mooel (team-based work, bui'ding-to-order, total commitment to 
quality, and worker ir./olvement in the work process, etc), he was especially 
interested in workers' reports that participation in the Aerovox education 
program helped them to work better in te?ms. He suggested that future 
education programs might focus more on skii's like team- building and on 



changing the organization of the company in general, as well as on more- 
traditional basic skills. The evaluation team meeting provided the venue for the 
vie* president for quality control to discuss this new Idea with other key people 
m the company and to begin to plan how these new Ideas might be carried out. 

Attached data-gathering instruments, summaries of findings, and related 
documents: 

Data-gathering instruments: 

- Job-Specific Questionnaire: same for ail students 

- "ASTD" Categories Questionnaire: same for all students 

- Competencies Checklists: tailored for 

• SPC Math 
•GEO 

• ESP (English for Special Purpose) 

• ASE/Pre ASE 
•ABE 2 

- Questionnaire for Department Heads 

Summary of findings: 

- Outcome Study Final Report 



EVALUATION REPORT- WORKPLACE LITERACY PROJECT 
SOUTH EASTERN MASSACHUSETTS UNIVERSITY (NOW KNOWN AS 
UNIVERSITY OF MASSACHUSETTS, DARTMOUTH) 
LABOR EDUCATION CENTER AND AEROVOX, INC. 
IN EFFECT FROM APRIL 1990 TO SEPTEMBER. 1991 

INTRODUCTION: 

v BACKGROUND: 

Initialy Aerovox was reluctant to release workers or management to make up an evaluation team 
or to take part In meetings outside the plant. The company considered that Hs safisfactton with the 
pro-am was proof of Its su^ 

However It acceded to u DOE request, and in the last months of 1990, an evaluation team was 
formed at Aerovox. 

As weB as the coordinator of the Workplace Literacy Program, It induded the vice President of 
Human Resources, the Person* ' Manager of the Electrical Products Group, the Production 
Manager of the Electrical Products Group, me Department Head of 54 workers Involved In the 
Workptace Literacy Program and a winder and a riveter who were students (one In ESP arxl 
Bi .?d t !! e Pr9 ^ s ^ a*** AS 5 cfess) to the program. At the Mass, Education Workplace 
Literacy Initiative conference of represemative teams from companies taking part In the 
evaluation, heW in Boston in January, 1991, the Aerovox team was represented by the VP of 
Human Resources and the coordmatorof the Workplace Literacy Program 
Following the meeting, the team had a series of meefjngs-often held between the WP Lit. 
coonSnator and irxfvidual team members, sometimes held as a group, and, on occasion, held as 
a group with Laura Sperazl. The coordinator and the VP also had correspondence and discussion 
with Laura. We studedttw original revised it to suit our 

needs and butt up an Aerovox work related skifis questtonaire for workers and supervisors. As we 
worked together, our goafs became more apparent, our enthusiasm grew and our belef in the 
efficacy of our evaluation strengthened. In October 1991 , we held a meeting to report to 
representatives of management. We limited our report to findings on Work Related Skills In which 
Department head responses were cross tabulated with worker/student responses. One of the 
responses from the findings 

6. "Workers are developing sense of team work and team identity 
stayed the same improved Improved a lot 

w/§ 1&2 % 6&5 % 284 % 

DH/f 445 % S&B % % 

triggered a response from the Vice President of Qualty Control which delighted the Evaluation 
team. "This," he said, stabbing the response with his finger, "This Is the response I am most 
Interested in. I have recently visited a plant in which worker involvement and problem solving were 
firmly established. The workers are the ones with the greatest understanding of the problems and 
the workers are the ones who can find the best solutions. We have relied too much on 
management and it is time for us to turn to the people on the floor." 



b. THE EVALUATION QUESTIONAIRES 

Y/orker /students drawn from five (Afferent classes completed three questtonalres. 

1. Work related SkBs 

2. ASTD -drawn from: 

i. Foundation (knowing how to team) 

H. AdaptabiBty(crealivemir^ 

i. Personal Management (self esteem, goal setting, motivation and personal and career 
advancement) 

iv. Group Efectiveness (interpersonal skffls, negotiation and teamwork) 

v. Group Effectiveness (organization skilfs and leadership) 

3. Competence (readftg writing and computation) was initially pari of the ASTD 1st, but it was 
withdrawn from this and a series of third questkmaires were drawn up by teachers and coonSnator. 
Responses were eicited from five Different classes 

I. English for Special Purposes (ESL classes with emphasis on job related language). 
Responses came from 20 students who were divided Into two groups - 

(a) ESP 1 (Beginners) & (Pre-lntermedate) — 2 cycles completeoVthe first of 48 hours, the 
second of 24 and 

(b) ESP 2 (Beginners )~with one competed cycle of 26 hours. 

2. Aduft Education 1 (more properly Adult Basic Ed.) 

Responses from 12 students who had completed one 12 week cyde of 48 hours or two 12 week 
cycles of 48 hours or three 12 week cycles of 48 hours 

3. Aduft Education 2 (more property Pre* Adu8 Secondary Education and Adult Secondary 
Education) Responses from 5 students who had completed one 1 2 week cycle of 48 hours or 
two twelve week cycles of 48 hours or three 12 week cycles of 48 hours, ( and who, at some 
stage, might have moved from Ad. Ed 1 to Ad. Ed 2) 

4. GEO . Responses from 9 students who had completed a cyde of 8 weeks and 32 hours (GED 
Express) or Pre-GED (12 weeks and 48 hours) or Pre-GED (12 weeks and 48 hours) plus GED (12 
weeks and 48 hours) 

5. Pre-SPC (Statistical Process Control ) Math workshops. Responses from 9 students from the 
first group who completed the 6 week and twelve hour workshop. 

Supervisors were asked to complete a Work Related Skills assessment and to comment on their 
attitudes to the Weracy program. The Work Related Skills assessments were cross tabulated with 
worker/a* jdent assessments. 

Worker /Student and Department Head/supervisor responses were entered on SPSS at UMass, 
Dartmouth and cross tabulated by department and classes. 
ASTD responses were entered. 

Competency responses were entered and cross tabulated with departments. 

Other more personal information has been entered -we Intend to mmake more cross tabulations. 

The self evaluation report was passed out to 83 students, 56 completed and returned it and 55 
were used In the evaluation. Ten were discarded because of faulty /inadequate responses. 

9 department heads or supervisors completed the Work Related Skills report. 



FINDINGS: 

A. WORK RELATED SKILLS 



(■) worker/students 

-w/» 



DH/I 

1 /Workers are not afraid to ask questions about things they dont understand 



stayed the same 
w/s 9 % 
DH/S 33% % 



improved 
99 % 
55 % 



improved a lot 
32 % 
U % 



improved a lot 
17 % 
112 % 

improved a lot 
17 % 
% 



2. *Workere understand directions 
stayed the same improved 
w/i 17 % 68 % 
EH/i 344 % 6&6 % 

3. *Workers figure out ways do things themselves 
stayed the same improved 
w/i 20 % 63 % 
DH/i 77JB % 212 % 

4. *Workers discover and apply new/better ways to do things at work 
stayed the same improved improved a lot 
w/i 18 % 37 % 15 % 
n3/ft 6&9 % 1L1 % % 

5. •Workers work well with co-workers 

stayed the same improved improved a lot 

w/f 30 % 48 % 22 % 

EH/i B&6 % 3X4 % 11 % 

6. *Workers are developing sense of team work and team identity 
stayed the same improved improved a lot 
w/i 182 % 6&5 % 164 % 
DH/i 448 % 5&S % % 
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7. "Workers can describe their jobs 

stayed the same improved improved a lot 

w/s 1&2 % 636 % IB4 % 

na/f objb % 4«a% % 

8. *Workers can relate their Jobs to overall production of capacitors at Aerovax 



improved 
67 % 
% 



stayed the same 
w/s 18 % 
SfeVs 77JB % 

9. *Workers inspect parts of their equipment 
stayed the some fmproued 
w/i 345 % 401 % 
DB/s 444 % 5S0 % 

10. •Workers practice maintenance of their equipment 
stayed the some improved 

w/i 327 % 8B4 % 

DH/i B&6 % 333 % 



tmproued a lot 
3D % 
% 

improved a lot 
104 % 
% 

improved a lot 
109 % 
1U % 



1 1. •Workers practice cleanliness of their equipment/surroundings 



stayed the some tmproued 
w/s 236 % 6L8 % 

DH/i 778 % 222 % 

12. •Workers take safety precautions 
stayed the some improved 
w/s 182 % 6&5 % 
DB/i 333 % 8&7 % 

13. *Workers understand need for Quality Control 
stayed the same improved 

w/s 13 % 07 % 

DB/s B&6 % 444 % 



improved a lot 
145 % 
% 

improved a lot 
164 % 
% 

tmproued a lot 
20 % 
% 



14. *Workers understand the purpose of having an MPI 
stayed the same Unproved 
wM 438 % 43j0 % 
TBjM 444 % S5j6 % 

15. *Workers can follow their MPI 
stayed the same improved 

wm ana % xa % 

TB/M 444 % 806 % 

16. •Workers can follow MFIs from other sections 
stayedthesame improved 
w/s 864 % 38.4 % 
DH/s 807% 303 % 

17. •Workers can read graphs around the plant 
stayed the same improved 
w/s 802 % 3C8 % 
EH/i 807 % 3&3 % 

18. •Workers can construct a graph 
stayed the same improved 
w/s 6UB % 32.7 % 
VH/t 883 % 1L1 % 

19. *Workers can understand SPC procedures at Aerovox 
stayed the same improved 
w/s 609 % 453 % 
DH/S 88JB % 44.4 % 

20. •Workers could take part in SPC procedures if asked to do so 

stayed the same Unproved improved a lot 

w/s BOZto 4O0 % US % 

HH/s BOB % 4C4 % 



improved a lot 
127 % 
% 

improved a lot 
127 % 
% 

improved a lot 
7S % 
% 

improved a lot 
73 % 
% 

improved a lot 
SJS % 
% 

improved a lot 
3S % 
% 
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21. "Workers understand Right to Know 

stayed the scone improved improved a lot 

w/i 3&2K 8BL2% 36 % 

m/B 44.4% «&% % 

OBSERVATIONS: 

Responses from Department head/supervisors compared to Worker/Student Responses. 

Student/worker responses were more positive man those of department heads/supervisors. I 
beleve that there may be several reasons for this. 

One rnay be Bravery nature of the floors work on tight 

schedules and the department heads are respons&te tor for a large number of workers and have a 
production expectation to meet. Workers at Aerovox are experienced and competent-most have 
had years on the ftoor8Jidrea(^ a deg^ of expertise that no workplace iteracy or any other 
program would change. 

Department heads responsJbte for large numbers of workers would see the same competent 
workereonthelneanditwouldbedrffkxjSfort^^ 

workers related their jobs to overaS production at Aerovox. Many of the jobs that our 
woffcerys&jdeflts are doing at present do not reflect the* skifis that workers are (earning in the 
classroom. These sWBs are preparing students 

recajirmg a wider range of skills, abilty lo deal with the new MPIs and Afferent QC (inducing SPC) 
methods plus the opportunity to trouble shoot and probiem sotvs. 
Finally, what we should probably have included in this assessment, is the change In the morale 
and company involvement in the workers involved in the program. This was pointed out by the 
Production Manager who said: "Workers come to the plant in better spirits; their days hold more 
for them than they dkt before Workplace Literacy. 

Some interesting exceptions to Department Head /Worker response discrepancies were: 
(a) Questions about MPIs 

Here the dept.heaoVsupervisor response Is either more positive than student response, or 
almost Identical. 

•Workers understand the purpose of having an MPI 

stayed the same improved improved a lot 

w/i 438% 43j6 % 127 % 

TB/B 44* % BBjB % % 

•Workers can follow their MPI 

stayed the same improved Unproved a lot 

w/i 0O9 % 9BA % 127 % 

EB/a 44* % SSjB % % . 



•Workers can follow MPIs from other sections 

stayed the same Unproved improved a lot 

w/i 9BA % 304 % 73 % 

DB/f 68.7 % 33L3 % % 

(b) Questions regarolng SPC procedures & Right to Know 

Here again, dept JieaoVsupervisor responses are more positive or virtually agree with 

worker/student responses. 

•Workers can understand SPC procedures in the plant 

stayed the same improved improved a lot 

w/i 009 % 4&5 % 36 % 

raft S&B % 444 % % 

•Workers could take part in SPC procedures If asked to do so 

stayed the same Unproved improved a hot 

w/i oaa % 400 % lb % 

DH/s BOB % 444 % 

•Workers understand Right to Know 

stayed the same improved improved a lot 

w/i aaa % s&2 % as % 

EB/t 444 % BSD % % 

The statements 

•Workers can understand SPC procedures In the plant 

•Workers could take part In SPC procedures If asked to do so 

to which only 49.1 & 41 .8 % of students responded as having improved is a reasonable % age. 
Engfsh for Special Purposes classes and the Express GED classes have not yet included SPC 
procedures in their lessons. 

Some of the job specific skills could not be taught in some classes. 
For example: 

Pre-SPC Math classes which are of 12 hours duration, do not go into reading and understanding 
of MPIs. The emphasis Is on acquiring math skills that wit! enable them to understand SPC 

English for Special Purposes classes have lessons bultt around understanding MPIs but have 
math classes that concentrate on understanding math terms and problem solving in English and 
do not yet aspire to more advanced math. 



The data which cross references the responses from Different departments shows many 
examples 8ke the following: 

•Workers practice maintenance of their equipment 

stayed the same improved improved a lot 

w/» 327 % 034 % 209 % 

TB/B BBS % 333 % 1X1 % 

Department 27 which fnctudes both MPP Assembly and MPP Wincing states that 32 % remained 
the same, 55.9 % Improved and that 1 1.8 % improved a tot. 

The 32% of students who fe» that they remained the same, came from the more necessarily 
stringent Wincing Department; those who felt that they had improved, came from the assembly 
Ine. I woukJ have been dteheartened by the survey, if i had heard those from MPP Winding claim 
that they were now taking better care of their equipment i would not have beleved their 
responses. 

I think that we should take particular notice of the following figures: 
•Workers take safety precautions 

stayed the same improved improved a lot 

w/» 182 % 65JS % 164 % 

TB/u 333 % 68.7 % % 



This is an encouraging response and one that can be substantiated by the following graph which 
appeared in the September 1991 edition of the Aerovox Newsletter 



September 199: 
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Corporate Safety Trends 

1989 1990 1991 1989 1990 1991 




lost T«ne Accidents Lost Time Days 

9 1991 Statistics through August 



I beleve that one of the essentials of successful workplace Iteracy programs is that senior and 
middle management, workers and educators agree on the goals of the program and work 
together to accompish these goals. Aerovox has been congratulated for the strength of the 
support that management has given to the program (.Including the hours of company time which 
have been devoted to ft) and on many occasions there has been praise given to the production 
managers and department heads. The congratulations were certainty earned, the praise most 
highly deserved. I should Ike, though, to bring before you an example of workplace Iteracy at Its 
best. 

I do not want to name these people but there Is one worker/student who is learning EngBsh for 
Special Purposes. This worker has a supervisor and another older worker who take a warm Interest 
in her. These women speak to her in Engish; they help and correct hen they check on what she is 
doing in class and follow ft up with her. When one of them filled in the supervisors report, she 
beamed. "We're so nappy for her." she said. The following is a breakdown of their separate 
answers. 

Both supervisors and worker/student answered: 

stayed the same to questions on 3,4,8, 11,1 6-21 . 9 questions that both agree 

have not yet been covered in the 26 hours of 
classes 

impmxd to questions on 1,2,6,8,11-14 

While only supervisors gave an 

improved a lot to questions on Workers work we!! with co-workers 

& Workers practice mainteance of their * 
equipment 



OVERALL TRENDS 

When department head/supervisor responses were examined It was apparent that d.h./s.who 
have w/s from the less skileo? bracket (that is those who are part of the ESP, Adult Ed 1& Pre 
SPC math classes) saw more improvement In workers In their Departments/ These dh/s came from 
Dept 27 (MPP Assembly), Dept. 25 (Sub Assembly) and Dept. 21 (Assembly) 

Other Department heads and supervisors whose departments are made up of 'more sophisticated 
• workers saw no change In their work related skills. These DH/Sups come from the Electronics 
section of Aerovox and Depts 20 (Winding), 
27 (MPP Winding), 22 (Final Test) , Shipping and Maintenance. 

Responses from student/workers reflected the same trend. 

Those who attended QED and Adult Ed 2 classes made up 50-100% of an the 'stayed the same" 
responses. In retrospect it is obvious that we should have included a 4th response to questions 
that indicated that student/worker was already in possession of the skill before classes began.. 

There may also be a matter of timing. GED (and many Adult Ed 2) students had completed classes 
long before the survey took place. Possibly their perceptions are more realistic but I am inclined to 
think that they did not bring the same thought to their responses as that brought by students who 
took class time to sit down and consider their answers. 
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FINDINGS of Department Head/Supervisor answers to questions on Workplace Ut Program 

I00%of aJ DH/supervlsors actively encourage students to take part In the program 
100% of aB DH/supervlsors understand the program in place 
100% of al DH/supervisors keep track of errors and who made them 

33% of DH/Supervisors stated that their production was OFTEN disrupted by students coming to classes 
33% of DH/Supervisors stated that their production was SOMETIMES disrupted by students coming to 
classes 

44% of DH/Supervisors stated that their production was SELDOM disrupted by students coming to 
classes 

The departments that stated that their production was OFTEN disrupted were Department 27 (Both MPP 
Assembly & Winding) and Department 22(FinaJTest). 

At the date of the survey , MPP Assembly had had 54 workers enrolled in classes. Despite her problems 
with production interruption, the Department Head has been one of the strongest supporters of the 
program. She has never refused permission for a student to leave the floor afihough at times she has had 
to imit the amount of time that several key operators could be away from the floor. This Department Head 
has also taken time to reorganize her Ines so that those whom we felt were most motivated and could most 
benefit from classes could leave for class. In common with an Department Heads, she has shown workers 
her pride and detght at their successes. 

MPP Winding and Final Test Department Heads have had fewer students enrolled in programs but these 
have often been highly skilled operators who were tfffteult to replace. At all times, however, they have 
given motivated students every encouragement to take part. 

78 % of all DH/Supervisors believe that they should identify areas needing Improvemnt in the 
program while 22% do not. 

One improvement was suggested and that was that even more emphasis should be placed on 
quality training. 



FINDINGS 

B. ASTO Questions 



Questions that were included in Work Related Skflb (reported tbore) hro been 

1) 



WORKgR /8 T DiaW r HE 8F0 WSB AS FOLLOWS 
L Foundation — knowing how to learn 

•I like to learn 

stayed the same improved improved a lot 

5% 67% 28% 

The five % who stayed the same camt from three successful graduates from the Express 

GED class. These students had liked to learn' all their Uvea 

•I know where to go for information to answer my questions (to a library, to a person) 

stayed the same improved improved a lot 

20% 63% 17% 

In our classes we have worked towards individual goal setting and study skills. The 20% 
who state that they stayed the same came from the fast paced GED Express class or 
from those who competed the first cycle in my own ESP classes which . I regret to say, 
must imply that some of the students were over dominated by their teacher. 

n. Communication — listening and oral communication 

•I listen to my co-workers' questions 

stayed the same Improved improved a lot 

20% 54% 26% 

Half of the 20% who stayed the same were again from the fast paced GED Express Class. 

I think that this was a skill already firmly in place for the rest of the 20% 

•I listen to my family memb ers - -wife / husband / children /other family/friends 

stayed the same improved improved a lot 

17% 66% 17% 

Again, half of the 20% who stayed the same were from the last paced GED Express 

Class. I think that this was a skill already firmly in place for the rest of the 20% 

•I ask questions clearly 

stayed the same improved improved a lot 

26% 61% 13% 

The more sophisticated' classes (that is GED. and Adult ED 2) made up the 26% who 
stayed the same. When I examine the responses I believe this to be true for many of the 
stayed the somes. These students were already confident of this skill. 



•I give explanations 
stayed the same 



improved 



improved a lot 



24% 65% 11% 

As before the "more sophisticated* classes (that Is GED, and Adult ED 2) made up the 26% who 
stayed me same. When I examine the responses I believe this to be true for many of the 
stayed the sames. These students were already confident of this skill. 

m. Adaptability-creative thinking and problem solving 
•I think about new/better ways to do things at work 



stayed the same 



improved 



improved a lot 



14% 66% 20% 

The stayed the sames came from wor ke rs in positions of high skills in departments 
with highly honed procedures. One of these students however recently received an 
Aerovox award as a problem alert winner. 

IV. Personal Management - self esteem, goal setting, motivation, and personal and 
career development 



•I value my opinions and my contributions 
stayed the same improved 



improved a lot 



22% 61% 17% 

I am cftpfldmt that our prog r am has done a great deal to enhance self confidence in 

worker students. 



•I offer my opinions to others 

stayed the same improved 

20% 72% 
As above. 

•I plan ahead In my personal life 

stayed the same Improved 

31% 51% 

We have emphasised personal goal setting in our classses.. 

•I plan ahead at work 

stayed the same improved 

30% 59% 

but we obviously need to have more discussion on it 



improved a lot 
8% 



improved a lot 
18% 



improved a lot 
11% 



o 
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"I am comfortable working atone 

stayed the same improved improved a lot 

31% 56% 13% 

The workers who stayed the same on this axe in positions in the company where they 
have had experience in working alone for years, I think that we could have rephrased or 
rethought this question, 

*I am a more active member of my community (ex Church, school groups, community 
groups, other) 

stayed the same improved improved a lot 

48% 46% 6% 

The Portuguese community is close knit and those of Portuguese descent are firmly 

entrenched members of their church and community groups. 

V. Group Effectiveness - interpersonal skills, negotiation and teamwork 

•I am willing to negotiate my needs and the needs of my group with my department head 

or group leader 

stayed the same improved improved a lot 

30% 5Q% u% 
Not yet. 

•I am supportive of others' efforts to Increase their skills 

stayed the same improved improved a lot 

24% 59% 17% 

The outreach from those already in the program to fellow workers has been one of its 
most warming aspects. An example could be taken from students to MPP Winding. They 
worked as teams towards their GED goal, studied together, made appointments for the 
exams together, comforted and encouraged those who had slipped below the pass score 
to try again. This team work has been in place for almost a year and their success rates 
have been a credit to it 

•I am supportive of others' learning efforts at hometexample: husband/wife, children, 
other family, friends) 

stayed the same improved improved a lot 

J 7% 70% 13% 

The response from workers who took the 'more basic' courses is reflected. They now 

have the confidence to their abilities to help their families. 
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vn. Influence — organizational effectiveness and leadership 

•I understand the long term goals of Aerovox and the essential part that education plays 
in making these goals a reality 

stayed the same improved improved a lot 

16% 54% 30% 

1 understand that it is possible forme to advance in the company as I increase my 
skills 

stayedthesame improved improved a lot 

13% 70% 17% 

OVERALL TRENDS 

Responses from the short term and mora advanced classes make up the largest proportions of 
stayed the somes . We should have had a preknowiege choice 
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FINDINGS 

B. COMPETENCIES - BY CLASS 

1. Competence Pte£PCliam Workshop 
Competence Hmlted to math skills as below 

Responses from 9 students from the first group who completed the 6 week and twelve hour 
workshop. 



•I am able to understand decimal notation 
stayed the some improved 

22% 67% 

•I am able to add whole numbers 

stayed the same improved 

11% 56% 

*I am able to add decimals 

stayed the same improved 

22% 45% 

•I am able to subtract whole numbers 

stayed the same improved 

0% 67% 

•I am able to subtract decimals 

stayed the same improved 

0% 78% 

*I am able to multiply whole nunbers 

stayed the same improved 

11% 33% 

•I am able to multiply decimals 

stayed the same improved 

11% 56% 

•I am able to divide whole numbers 

stayed the same improved 



11% 

*I am able to divide decimals 
stayed the same 

11% 



56% 

improved 
78% 



Improved a lot 
11% 

improved a lot 
33% 

improved a lot 
33% 

improved a lot 
33% 

improved a lot 
22% 

improved a lot 
56% 

improved a lot 
33% 

improved a lot 
33% 

improved a lot 
11% 



•I am able to change fractions to decimals 

improved 



stayed the same 
11% 

•I am able to read graphs 
stayed the same 

22% 

•I am able to graph 
stayed the same 

22% 

•I am able to find averages 
stayed the same 

11% 



78% 

improved 
56% 

improved 
67% 

improved 
67% 



improved a lot 
11% 

imprtved a lot 
22% 

improved a lot 
11% 

Improved a lot 
22% 



•I am able to use calculator-addltlon/subtractlon/multlplcatlon/dlvision (With or 
without decimals) 

stayed the same improved improved a lot 



11% 



5o% 



33% 



Here teacher assess me nt most closely paralled those of the students.The same criteria 
were used and she based her assessment on her pie- and post-workshop tests and her 
own observations. 



3» Competence? ffngflth far Spcc&sl Purposes 



ESP (ESL classes with emphasis on job related language). Responses came from 20 students 
who were Divided into two groups - 

(a) ESP 1 (Beginners) & (Pro-Intermediate) — 2 cycles compteted-the first of 48 hours, the 
second of 24 and 

(b) ESP 2 (Beginners )~wflh one competed cycle of 26 hours. 



improved a lot 
25% 



•I need less translation from my native language into English 
stayed the same improved 



0% 



75% 



•I am able to read signs and notices around the plant 
stayed the same Improved 



0% 

•1 am able to identify an MPI 
stayed the same 

25% 



75% 



improved a lot 
25% 



improved 
50% 



improved a lot 



(hay of the ESP classes had not progressed to the studying ofMPIs) 



25% 



•I am able to locate and use sections of MPI 
stayed the same improved 

25% 50% 
(hay of ESP dosses had not progressed to MPIs) 

•I am able to understand Aerovox graphs 
stayed the same improved 

25% 75% 
(Not aU classes had studied graphs) 



Improved a lot 
2596 

improved a lot 
0% 



Writing 

•I am able to fill out forms 
stayed the same 



improved 



12.5% 87.5% 

•I am able to fill out time tickets 

stayed the same tmproved 

0% 75% 
•I am able to fill out inspection sheets (if applicable) 
stayed the same improved 
0% 93% 



improved a lot 

improved a lot 
25% 

improved a lot 
7% 



•I am able to understand different math terms in English 
stayed the same Improved 



14% 



78% 



•I am able to do basic addition and subtraction 

stayed the same improved 

25% 50% 

•I am able to solve word problems 

stayed the same improved 

37% 50% 

♦I am able to estimate answers 

stayed the same improved 

30% 69% 



improved a lot 
8% 

improved a lot 
25% 

improved a lot 
13% 

improved a lot 
0% 



Teacher assessment: 

We should have used different competencies for theseo groups. Obviously those who had 
completed 2 cycles (with 72contact hours) gave more positive responses than those who 
had completed one cycle (26 contact hours) But a camparUson with teacher assessment 
records paraHed student assessment in both groups. 
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3. Competen ces GED 



. QED . Responses from 9 students who had completed a cycle of B weeks and 32 hours (GED 
Express) orPre-GED (12 weeks and 48 hows) or Pre-GED (12 weeks and 48 hours) plus GED (12 
weeks and 48 hours) 



•I am able to use given or remembered ideas in a different context from the one provided 
stayed the same ' 



66% 

•I am able to restate information 
stayed the same 

67% 

•I am able to summarize ideas 
stayed the same 

67% 



improved 
23% 

Improved 
22% 

improved 
22% 



•I am able to identify ideas that are implied 
stayed the same improved 



56% 



23% 



*I am able to read manufacturing process instructions 
stayed the same improved 



56% 



33% 



•I am able to draw conclusions from articles 
stayed the same improved 



56% 



33% 



•I am able to distinguish facts from opinions 
stayed the same improved 



67% 



11% 



fmproved a lot 
11% 

improved a lot 
11% 

improved a lot 
11% 

improved a lot 
11% 

improved a lot 
22% 

improved a lot 
11% 

Improved a lot 
22% 



•I am able to distinguish a conclusion from supporting statements 

stayed the same improved improved a lot 



56% 



44% 



*I am able to distingui s h cause and effect relationships 
stayed (he same improved 



56% 



44% 



*I am able to make judgments about the accuracy of information 
stayed ihe same improved 



56% 



44% 



0% 

improved a lot 
)% 

improved a lot 
0% 
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Writing 



A . Writing Skills 

•I can identify and correct sentence fragments 
stayed the same improved 



33% 



67% 



•I can Identify and correct run-on sentences 
stayed the same improved 



22% 



56% 



•I can identify and correct comma splices 

stayed the same improved 

22% 56% 

*I can check subject/verb agreement 

stayed the same improved 

22% 78% 

•I can check verb tense agreement 

stayed the same improved 

22% 78% 

*I can correct capitalization 

stayed the same improved 

22% 78% 

•I can correct punctuation 

stayed the same improved 

33% 56% 



improved a lot 

improved a lot 
22% 

improved a lot 
22% 

improved a lot 
0% 

improved a lot 
0% 

improved a lot 
0% 

improved a lot 
11% 



B, Writing an Eftsay 



•I am able to form and state an opinion 
stayed the same improved 

44% 45% 

•I am able to support my opinion 

stayed the same Unproved 

33% 45% 

•I am able to form and state an explanation 
stayed the same improved 



22% 



67% 



•I am able to back up my explanation 

stayed the some improved 

44% 45% 

•I am able to state a conclusion 

stayed the same improved 

33% 57% 

•I am able to write a clear essay 

stayed the same improved 

44% 56% 

■I am able to write a straightforward letter 
stayed the same improved 



33% 



67% 



improved a lot 
11% 

improved a lot 
22% 

improved a lot 
11% 

improved a lot 
11% 

improved a lot 
11% 

improved a lot 
0% 

Improved a lot 
0% 
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*I am able to read and solve word problems 
stayed the same improved 

11% 785 

•I am able to read graphs and tables 



stayed the same 
56% 



improved 

22% 



•I am able to use formulas for perimeter 
stayed the same improved 

44% 34% 
•I am able to use formulas for area 



stayed the same 
33% 



improved 
45% 



•I am able to use formulas for volume 
stayed the same improved 



11% 



56% 



•I am able to use formulas for circumference 
stayed the same improved 



22% 



675 



•I am able to use formulas for distance and cost 
stayed the same improved 



22% 



56% 



•I am able to use ratio and proportion to solve word problems 
stayed the same improved 



22% 



56% 



•I am able to understand probablility. mean, median 
stayed the same improved 



22% 



67% 



improved a lot 
11% 

improved a lot 
22% 

improved a lot 
22% 

improved a lot 
22% 

improved a lot 
33% 

improved a lot 
11% 

improved a lot 
22% 

Improved a lot 
22% 

improved a lot 
11% 



*I am able to use basic algebra to solve word problems 

stayed the same improved improved a lot 

22% 67% 22% 

•I am able to use formula for finding the slope of a line 

stayedthesame improved Improved a lot 

22% 675 22% 

•I am able to plot and find points on an x, y axis 

stayedthesame improved improved a lot 

22% 565 22% 

1 amable to use Pythagorean relationship to solve hypotoneuse problems w/ triangles 
stayedthesame improved Improved a tot 

22% 56% 22% 

•I am able to understand angle relationships 

stayedthesame improved improved a lot 

78% u% 

•I am able to measure angles of a triangle at d solve similarity problems 
stayedthesame improved improved a lot 

0% 78% 22% 

Teachor assessment: 

Here comparison of teacher and student assessments showed that there were virtually parallel 
assessments In Math (although the teacher found fewer stayed the sames ) There were 
discrepancies In the comparisons between reading and writing assessments where teachers 
assessed greater Improvement Many of these criteria are, of course, judged more subjectively 
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4. Competence; Pre - Adult Secondar y Education and FiMdrit Secondly Education / 
ASB (Adult Fducitlon 2) 

3. Aduft Education 2 . Responses from 5 students who had completed one 12 week cycle of 48 
hours or two twelve week cycles of 48 hours or three 1 2 week cycles of 48 hours, ( and who, at 
some stage, mlQht have moved from Ad. Ed 1 to Ad. Ed 2) 

Beadfatf 

•I am able to answer T/F questions about work materials 

stayed the same improved improved a lot 

80% 20% 

*I am able to answer T/F questions about local newspaper articles 

stayed the same improved improved a lot 

20% 80% 

•1 am able to answer multiple choice Questions about work materials 

stayed the same improved improved a lot 

20% 80% 

•I am able to answer multiple choice questions about articles from local newspapers 
stayed the same improved improved a lot 

20% 80% 

1 am able to use context to work out the meanings of words 

stayed the same improved improved a lot 

40% 40% 20% 

*l am able to use some root words/prefixes or suffixes to work out meanings of words 
stayed the same improved improved a lot 

20% 80% 

*I am able to use a dictionary 

stayed the same improved improved a lot 

40% 20% 80% 
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Writing 



1 am able to capitalize 
stayed the same 

20% 



improved 
4096 



•I am able to punctuate using capitals and periods 
stayed the same improved 

80% 

•I am able to punctuate questions, commands and explanations 
stayed the same improved 



20% 

•I am able to Identify nouns 
stayed the some 



80% 

improved 
60% 



"I am able to use pronouns correctly 

stayed the same improved 



20% 



80% 



*I am able to use different verb times 

stayed the same improved 

100% 

*I am able to make subject /verb agreement 
stayed the same improved 

100% 

•I am able to identify sentence fragments 
stayed the same improved 

100% 

•I am able to Identify run- on sentences 
stayed the same improved 



20% 



80% 



*I am able to identify comma splices 

stayed the same improved 



improved a lot 
405 

improved a. lot 
20% 

improved a lot 



improved a lot 
40% 

improved a lot 



improved a lot 



Improved a lot 



improved a lot 



improved a lot 



improved a lot 



40% 



60% 
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•I am able to write clear sentences 

stayed the same improved 

100% 

•I am able to combine sentences using connecting words 
stayed the same improved 

100% 

*I am able to write a short summary 

stayed the same improved 

100% 

•I am able to write a short paragraph 
stayedthesame improved 

100% 



improved a lot 



improved a lot 



improved a lot 



improved a lot 
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•I am aware of all symbols defining operations 

stayed the same improved tmproved a lot 

10(3% 

•J am able to read and solve word problems in addition using whole numbers 
stayedthesame improved Improved a lot 

100% 

•I am able to read and solve word problems in subtraction using whole numbers 
stayedthesame Improved tmproved a lot 

100% 

*I am able to read and solve word problems in multipication using whole numbers 
stayedthesame unproved tmproved a lot 

100% 

•I am able to read and solve word problems in division using whole numbers 
stayedthesame Improved tmproved a lot 



100% 

1 am able to understand decimal notation 



stayedthesame 



•j bxu sbitc to &dd doctm&Js 
stayedthesame 

•I am able to subtract decimals 
stayedthesame 



•I am able to multiply decimals 
stayed the same 



•I am able to divide decimals 
stayedthesame 



improved 
100% 
improved 
100% 
improved 
100% 
improved 
100% 
improved 
100% 



tmproved a lot 



tmproved a lot 



Improved a lot 



improved a lot 



tmproved a lot 



1 am able to read and solve word problems In addition using decimals 
stayedthesame tmproved tmproved a lot 



60% 



40% 



*I am able to read and solve word problems in subtraction using decimals 
stayed the same improved tmproved a lot 



80% 



20% 



1 am able to read and solve word problems in multipication using decimals 
stayedthesame improved tmproved a lot 



80% 



20% 



•I am able to read and solve word problems in division using decimals 
stayed the same 



•I am able to add fractions 
stayed tiie same 

1 am able to subtract fractions 
stayed the same 

1 am able to multiply fractions 
stayed the same 

•I am able to divide fractions 
stayed the same 



•I can get information from graphs 
stayed the same 

20% 

*l can get information from tables 
stayed the same 

20% 

1 am able to plot an SPC chart accurately 
stayed me same improved 



improved 
80% 

improved 
60% 

improved 
60% 

improved 
80% 

improved 
80% 

improved 
60% 

improved 
80% 



improved a lot 
20% 

improved a lot 
40% 

improved a lot 
40% 

improved a lot 
20% 

improved a lot 
20% 

improved a lot 
20% 

improved a tot 
improved a lot 



40% 60% 

*l understand what to do when SPC readings are not within the tolerance 
stayed the same improved improved a lot 



20% 

Teacher, jtannm e nts 



80% 



Teachers felt that there had been more improvement In reading and writing and that i 
percentage of of the math skins had improved a lot 



9 ~» 



Competence; Adnlt Basic Education (Adult Education 1) 



AduB Education 1 

Responses from 1 2 students who had completed one 12 week cycle of 48 hours or two 12 week 
cycles of 48 hours or three 12 week cycles of 48 hours 



*I can figure out the meaning of new words by reading the whole sentence and/or the 
whole paragraph 

stayed the same improved. Improved a lot 



9% 



91% 



•I can answer questions about what I have read 
stayed the same Unproved 

91% 



improved a lot 
9% 



•I have strategies for reading (ex. I can understand new words by looking at the whole 
sentence or paragraph .) 

stayed the same improved improved a lot 



9% 



73% 



18% 



*l know its most important that I understand the whole story and its OK if I don't 
understand a word or two 

stayed the same improved improved a lot 



82% 



18% 



*l have strategies for answering questions about a story. When I answer a question. I go 

back to the story to make sure Fm getting the right answer 

stayed the same Improved improved a lot 



91% 



9% 



•I am sure that reading is a matter of practicing. The more I do-the better I get 
stayed the same 



• I read more 
stayed the same 

9% 

* I can understand bar graphs 
stayed the same 



improved 
82% 

improved 
64% 

improved 
82% 



improved a lot 
18% 

improved a lot 
27% 

improved a lot 
18% 



•I can read a US Map 
stayed the same 

9% 



improved 
73% 



improved a lot 
18% 
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•I know about US geography 
stayed the same 

36% 
Writing 



improved 
37% 



Improved a lot 



•I know that writing is a matter of practice— the more I do it-the better I get 
stayed the same 



•I can write sentences 
stay ed the same 



♦I can organize my sentences 
stayed the same 

9% 



improved 
91% 

improved 
82% 

improved 
82% 



improved a lot 
9% 

improved a hot 
18% 

improved a lot 
9% 



*l name the most important idea first, then the second most Important one etc 
stayed the same improved 



9% 

•I can write a short summary 
stayed the same 



82% 

improved 
91% 



*I can write a short paragraph (in response to a question] 
stayed the same improved 



9% 



82% 



improved a lot 
9% 

improved a lot 
9% 

improved a lot 
9% 



Computation (As Cor Pre-ASC-ASE or Adult Ed2) 
*I am aware of all symbols defining operations 
stayed the same improved 



improved a lot 



18% 73% 9% 

* l a^abte to read and solve word problems in addition using whole numbers 

stayed the same improved improved a lot 

64% 46% 

I amable to read and solve word problems in subtraction using whole numbers 
stayed the same improved improved a lot 

64% 

I am able to read and solve word problems in multipication using whole numbers 
stayedthesame improved improved a lot 

64% 36% 
n am able to read and solve word problems in division using whole numbers 
stayed the same * • 



improved 
64% 



•I am able to understand decimal notation 
stayed the same 



n am able to add decimals 
stayed the same 



•I am able to subtract decimals 
stayedthesame 



n am able to multiply decimals 
stayedthesame 



*I am able to divide decimals 
stayedthesame 



improved 
73% 

improved 
64% 

improved 
64% 

improved 
64% 

improved 
73% 



improved a lot 
36% 

improved a lot 
27% 

improved a lot 
36% 

improved a lot 
36% 

improved a lot 
36% 

improved a lot 
27% 



*I am able to read and solve word problems in addition using decimals 
stayedthesame improved improved a lot 



73% 



27% 



*l am able to read and solve word problems in subtraction using decimals 
stayedthesame improved improved a lot 



73% 



27% 



I am able to read and solve word problems In multipication using decimals 
stayedthesame Improved improved a lot 

73% 27% 

I am able to read and solve word problems in division using decimals 
stayed the same improved improved a lot 



73% 



27% 
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*I am able to add fractions 



ctm xArf gjumn 
SuXjfuEL WE SCuTW 


If f ijif vvcu 


iffiwffwiiflgf a lnt 

If ftyfl t/vcriA 14 Ufl 






36% 


i am aoic to suotnict xracuons 










it f^#f l/l«U 1* US* 




64% 


36% 


1 ttlU fUJIC M# tUUlUjSlj Ai a w UU 1 129 






stoned th& some 


imnroved 


ffTiBrymgrf n 




73% 


27% 


*T aid flhlp to Httriri^ firagHoiMtt 










tmoTOved a lot 

illUrf WvU %S Prf 




64% 


36% 


1 Can {(CI UUUI iuaUOu uuul (Jiapio 






stayed the same 


improved 






54% 


46% 


•I can get information from tables 






stomal the some 


Improved 




9% 


55% 


36% 


•I am able to plot an SPC chart accurately 




stoj/ed (he some 


tmproued 


tmproued a lot 


9% 


62% 


9% 


•I understand what to do when SPC readings are 


not within the tolerance 


stayed the some 


tmproued 


tmproued a lot 


26% 


73% 


9% 



Teacher Assessment* 

Virtually parallel with student assessment 
OBSERVATIONS 

We spent some time putting together those competencies and discussed them with the 
students. While we were not entirely satisfied with them, we decided that we ware satisfied 
enough to use them for our goats and student pre and final assessment sheets for each cycle. 
We wiH continue to make adjustments and changes to them but think that they make a good basis 
tor individual and class records. Teacher assessments which had been made before and after 
each cycle were not always based on the evaluation criteria and were not fed into the SPSS 
program. When several dtfferent groups were included In the overall response, as was the case 
in each class with the exception of the Pre-SPC Math Workshops , the compared teacher 
pssessments were not always parallel with student assessments. This was particularly true of the 
GEO classes, when some of the students woutt claim that many of the skills were already in place 
and Just needed revision. In some cases this may be true and in some cases it might be Inked with 
pride. Which is just fine. 



SUGGESTIONS FOR FUTURE SURVEYS: 

1 .1 have suggested In my final Workplace Ueracy report ttiat In future after proposals have been 
received by the DOE that they should short 1st the candidates and that before the final grants are 
awarded, a representative from DOE should visit partners on the short 1st. The representative 
should ensure that each partner fully understands his resportsWRies and Is prepared to honor 
them. One of the issues that shouH be dtecussed Is that of evaluation-Its goals and the team 
work and company involvement in the process. Only after aB partners fully understand this and 
other Issues and an agreement has been reached, should the final grantees be chosen. I think 
Aerovox wfH vabe this evaluation and appreciate that we wiU be aJWe to use the results for aoss 
checking with company records on production, Quafty control and safety , but H would certainly 
have been an advantage If the evaluation process had been accepted at the outset. 

2. When responses have been coSected from an Inrtal evaluation l&e this one, it would be 
valuable if the initial data could be processed by a central consultant -preferably one hired by the 
DOE.We do not provide computer classes in the Workplace Lit. program at Aerovox and the 
systems used at UMass and in the plant were not compatible with the SPSS system available. 
When we finaSy got some of the data into place and received the resuRs, It was suggested to me 
that it would have been less time consuming if iti simply sat down with a calculator and a pen and 
pencil. This sounds rather defeatist , but it is true that while I very much approve of the ktea of 
computer analysis and believe that we can overcome the Difficulties that we had this time, I think 
that a central consultant would have been cost effective. 

3. We intend to make the survey an on-going process. Students and department heads will be 
asked for responses both before and after each cycJe.We will include the Pre*nowiedge option 
incur cjjestionafres. As I have already said, we are reviewing regularly and making the 
questtonaires part of the basis for goal and expectation setting. Teachers will be basing individual 
and class assessments on the competencies and some of the ASTD questions. 

4. 1 hope that future surveys will Include a researcher of the dedication and understanding of 
Laura Sperazf who made ail this possible. 



Self Report Checklist for Education Program Participants 
University of Massachusetts, Dartmouth & Aerovox, Inc. 
TeclmicslEngUshSkflbFro^sm 

Year of birth: Country of Origin: 

Gender M/F Marital Status: M/S/D/W 

(married, single, divorced, widowed) 

Number and ages of children: , 

Native language: 0 Portuguese 0 Korean Other 

Other languages: 

Language(s) used at home: _ 

Language(s) used at work: ,_ . 

Year of Arrival in US: . 

Occupation in country of origin: . — 

Occupation in US: 

Number of years at Aerovox: 

Education in country of origin: (from age to ) 

Education in US: No / Yes (last grade completed) 
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V. Personal Management -- self esteem, goal setting, motivation, and personal and 
career development 



•I value my opinions and my contributions 
stayed the same improved 

offer my opinions to others 
stayed the same improved 
*I plan ahead In my personal life 
stayed the same improved 
(If you wish, please say what your plans are:. 



improved a tot 



improved a tot 



improved a lot 



•I plan ahead at work 

stayed the same improved 

(If you wish, please say what your plans are:. 



improved a lot 



•I am comfortable working alone 

stayed the same improved improved a lot 

•I am a more active member of my community (ex. Church, school groups, community 
groups, other) 

Please name group/s — 

stayedthesame improved improved a lot 

VL Group Eflcctlrene— - interpersonal skills, negotiation and teamwork 
•I work well with my co-workers 

stayedthesame improved Unproved a lot 

•I am developing a sense of team work and team identity 

stayedthesame improved improved a lot 

•I am willing to negotiate my needs and the needs of my group with my department heud 
or group leader 



stayed the same improved 

•I am supportive of others' efforts to Increase their skills 

stayed the same improved 



improved a lot 



irriproved a lot 
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VUL Other Job-specific tasks or skills 

•I can describe my Job 

stayed the same Unproved 

• can relate my Job to the overall production of capacitors 

staved the same improved 

•I inspect parts of my equipment 

stayed the same improved 

•I practice maintenance of my equipment 

stayed the same Unproved 

•I practice cleanliness of my equipment/surrou n d in gs 

stayed the same Unproved 

•I take safety precautions 

stayed the scone improved 

•I understand need for Quality Control 

stayed the same improved 

*I understand the purpose of having an MPI 

stayed the same Unproved 

•I can follow my MPI 

stayed the same Unproved 
*I can follow MPIs from other sections 
stayed the same Unproved 
•I can read graphs around the plant 
stayed the same Unproved 
•I can construct a graph 
stayed the same Unproved 
•I can understand SPC procedures in the plant 
stayed the same Unproved 
•I could take part in SPC If asked to do so 
stayed the same Unproved 

•I understand Right to Know 

stayed the same Unproved 



Improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



1. Competence: Bute Math Workshop Competence limited to math skills as below 

•I am able to understand decimal notation 
stayed the same improved 
•I am able to add whole numbers 
stayed the same improved 
1 am able to add decimals 
stayed the same improved 
•I am able to subtract whole numbers 
stayed the same improved 
•I am able to subtract decimals 
stayed the same improved 
*I am able to multiply whole nunbers 
stayed the same improved 
•I am able to multiply decimals 
stayed the same improved 
•I am able to divide whole numbers 
stayed the same improved 
1 am able to divide decimals 
stayed the same improved 
•I am able to change fractions to decimals 



stayed the same 

•I am able to read graphs 

stayed the same 

•I am able to graph 

stayed the same 

•I am able to find averages 

stayed the same 



improved 



improved 



improved 



improved 



improved a lot 



improved a lot 



improved a lot 



Improved a hot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



impr oved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



•I am able to use calculator-additlon/subtxactlon/multlpicatlon/dlvlslo i (With or 
without decimals) 



stayed the same 



improved 



improved a lot 



0 

ERIC 



o • > ./ 

** Kr 



*I need less translation from my native language into English 

stayed the same improved 

•I am able to read signs and notices around the plant 

stayed Ore same improved 

•I am able to identify an MPI 

stayed the same improved 

*I am able to locate and use sections of MPI 

stayed the same improved 

*I am able to understand Aerovox graphs 

stayed the same improved 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



Writing 

•I am able to OH out forms 

stayed the same improved 

•I am able to fill out time tickets 

stayed the same improved 

*I am able to fill out Inspection sheets (if applicable) 

stayed the same improved 

Computation 

•I am able to understand different math terms in English 
stayed the same improved 
•I am able to do basic addition and subtraction 
stayed the same improved 

m 

•I am able to solve word problems 

stayed the same improved 

•I am able to estimate answers 

stayed the same improved 



improved a lot 



improved a lot 



improved a lot 



Improved a lot 



improved a lot 



improved a lot 



improved a lot 



Reading 

•I am able to use given or remembered ideas in a different contest from the one provided 
stayed the same improved improved a lot 

1 am able to restate information 
stayed the same improved 
•I am able to summarize ideas 
stayed the same improved 
*l am able to identify ideas that are implied 
stayed the same improved 
*I am able to read manufacturing process Instructions 
stayea the same improved 
•I am able to draw conclusions from articles 
stayed the same improved 
*l am able to distinguish facts from opinions 
stayed the same improved 
1 am able to distinguish a conclusion from supporting statements 
stayed the same improved Improved a lot 

•I am able to distinguish cause and effect relationships 

stayed the same improved improved a lot 

•I am able to make judgments about the accuracy of Information 
stayed the same Unproved unproved a lot 



improved a lot 



improved a lot 



Unproved a lot 



improved a lot 



improved a lot 



improved a lot 



Writing 

A. Writing Skills 

•I can Identify and correct sentence fragments 

stayed the same improved 

•I can identify and correct run-on sentences 

stayed the same improved 

*I can identify and correct comma splices 

stayed the same improved 

•I can check subject/verb agreement 

stayedthesame improved 

•I can check verb tense agreement 

stayed the same Improved 

•I can correct capitalization 

stayed the same Improved 

*I can correct punctuation 

stayed the same improved 

B. Writing an Essay 

•I am able to form and state an opinion 

stayedthesame unproved 

1 am able to support my opinion 

stayed the same improved 

•I am able to form and state an explanation 

stayed the same improved 

*I am able to back up my explanation 

stayed the same improved 

*I am able to state a conclusion 

stayed the same improved 

•I am able to write a clear essay 

stayed the same improved 

•I am able to write a straightforward letter 

stayed the same improved 



improved a lot 



improved a lot 



improved a hot 



improved a lot 



improved a lot 



improved a lot 



Improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



•I am able to read and solve word problems 
stayed the same improved 
* I am able to read graphs and tables 
stayed the same improved 
•I am able to use formulas for perimeter 
stayed the same improved 
*I am able to use formulas for area 
stayed me same improved 
•I am able to use formulas for volume 
stayed the same improved 
•I am able to use formulas for circumference 
stayed the same improved 
*I am able to use formulas for distance and cost 
stayed the same improved 
*I am able to use ratio and proportion to solve word problems 
stayed the same improved 
•I am able to understand probablillty, mean, median 
stayed the same improved 
•I am able to use basic algebra to solve word problems 
stayed the same improved 
•I am able to use formula for finding the slope of a line 
stayed the same improved 
*I am able to plot and find points on an x, y axis 
stayed the same improved 
*I am able to use Pythagorean relationship to solve hypotoneuse problems w/ triangles 
stayed the same improved improved a lot 

•I am able to understand angle relationships 

stayed the same improved improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



Improved a lot 



improved a lot 



improved a lot 
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•I am able to measure angles of a triangle and solve similarity problems 
stayed the same improved Improved a lot 



4. Competence Pie -ASE/ASE (Adult Education 2) 



•I am able to answer T/F questions about work materials 

stayed the same improved improved a lot 

•I am able to answer T/F questions about local newspaper articles 

stayed the same Unproved improved a lot 

•I am able to answer multiple choice questions about work materials 

stayed the same improved improved a lot 

*l am able to answer multiple choice questions about articles from local newspapers 

stayed the same improved improved a lot 

*I am able to use context to work out the meanings of words 

stayed the same improved improved a lot 

•I am able to use some root words/prefixes or suffixes to work out m^nnii^ga of words 



improved 



improved 



improved 



stayed the same 
* am able to use a dictionary 
stayed the same 
Writing 

•I am able to capitalize 
stayed the same 
*l am able to punctuate using capitals and periods 
stayed the same improved 
•I am able to punctuate questions, commands and explanations 
stayed the same improved 
*I am able to identify nouns 
stayed the same improved 
•I am able to use pronouns correctly 
stayed the same improved 
•I am able to use different verb times 
stayed the same improved 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 
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H am able to make subject /verb agreement 
stayed ite some improved 
•I am able to Identify sentence fragments 
stayed the scone improved 
•I am able to identify run- on sentences 
stayed the scone improved 
•I am able to identify comma splices 
stayed the scone improved 
*I am able to write clear sentences 
stayed the scone improved 
*I am able to combine sentences using connecting words 
stayed the same improved 
•I am able to write a short summary 
stayed the same improved 
•I am able to write a short paragraph 
stayed the scone improved 
Computation 

•I am aware of all symbols denning operations 
stayed the same improved 
•I am able to read and solve word problems in addition using whole numbers 
stayed the same improved improved a lot 

•I am able to read and solve word problems in subtraction using whole numbers 
stayed the same improved improved a lot 

*I am able to read and solve word problems in mulupication using whole numbers 
stayed the same improved improved a lot 

•I am able to read and solve word problems in division using whole numbers 
stayed the same improved improved a lot 

•I am able to understand decimal notation 

stayed the same improved improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



unproved a lot 



improved a lot 



•I am abk to add decimals 
stayed the same improved 
1 am able to subtract decimals 
stayed the same improved 
•I am able to multiply decimals 
stayed the same improved 
•I am able to divide decimals 
stayed the same 



improved a lot 



improved a lot 



improved a lot 



improved improved a lot 

•I am able to read and solve word problems in addition using decimals 

stayedthesame improved improved a lot 

•I am able to read and solve word problems in subtraction using decimals 
stayedthesame improved improved a lot 

*l am abk to read and solve word problems in multiplcatlon using decimals 

stayed the same improved improved a hot 

•I am able to read and solve word problems in division using decimals 



stayed the same improved 

•I am able to add fractions 

stayed the same improved 

•I am able to subtract tractions 

stayed! the same improved 

*l am able to multiply fractions 

stayed the same improved 

•I am able to divide fractions 

stayed the same improved 

•I can get information from graphs 

stayed the same improved 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



•I can get Information from tables 

stayed the same improved improved a tot 

•I am able to plot an SPC chart accurately 

stayed the same improved improved a lot 

•I understand what to do when SPC readings are not within the tolerance 
stayedthesame improved improved a lot 



Competence ABE (Adutt Education 1) 



*I can figure out the meaning of new words by reading the whole sentence and/or the 
whole paragraph 



stayed the same improved 
•I can answer questions about what I have read 
stayed the same improved 



improved a lot 



improved a lot 



*l have strategies for reading (ex. I can understand new words by looking at the whole 
sentence or paragraph .) 



stayed the same 



improved 



improved a lot 



•I know its most important that I understand the whole story and its OK if I don't 
understand a word or two 



stayed the same 



improved 



improved a lot 



*I have strategies for answering questions about a story. When I answer a question, I go 
back to the story to make sure I'm getting the right answer 

stayed the same improved improved a lot 

•I am sure that reading is a matter of practicing. The more I do-the better I get 



stayed the same improved 

• I read more 

stayed the same improved 

• I can understand bar graphs 

stayed the same improved 
*I can read a US Map 

stayed the same improved 

• I know about US geography 

stayed the same improved 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 
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Writing 

•I know that writing is a matter of practlcc-the more I do tt-thc better I get 
stayed the same improved improved a lot 

•I can write sentences 

stayed the same improved improved a tot 

•I can organize my sentences 

stayed the same improved improved a lot 

*I name the most important idea first, then the second most important one etc 
stayed the same improved improved a tot 

•I can write a short summary 

stayed the same improved improved a lot 

•I can write a short paragraph (in response to a question) 

stayed the same improved improved a lot 



Computation (As for Pre-ASE-ASE or Adult Ed2) 



*I am aware of all symbols defining operations 

stayed the same improved improved a lot 

•I am able to read and solve word problems In addition using whole numbers 
stayed the same improved Improved a Jot 

*l am able to read and solve word problems in subtraction using whole numbers 
stayed (he same improved " improved a lot 

1 am able to read and solve word problems in multlpicatlon using whole numbers 
stayed the same improved improved a lot 

*I am able to read and solve word problems in division using whole numbers 



stayed the same improved 
•I am able to understand decimal notation 
stayed the same improved 
•I am able to add decimals 
stayed the same improved 
•I am able to subtract decimals 
stayed the same improved 
•I am able to multiply decimals 
stayed the same improved 
•I am able to divide decimals 
stayed the same 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved 

•I am able to read and solve word problems in addition using decimals 

stayed the same improved improved a lot 

•I am able to read and solve word problems In subtraction using decimals 
stayed the same improved improved a lot 
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•I am able to read and solve word problems in multipicatlon using decimals 
stayed the same improved improved a lot 

*l am able to read and solve word problems in division using decimals 



Unproved 



improved 



improved 



improved 



improved 



improved a lot 



stayedthesame 
•I am able to add fractions 
stayed the same 
•I am able to subtract fractions 
stayedthesame 
*I am able to multiply fractions 
stayed the same 
•I am able to divide fractions 
stayed the same 

•I can get information from graphs 
stayed the same improved 
•I can get information from tables 
stayed the same Improved 
•I am able to plot an SPC chart accurately 
stayed the same improved 
1 understand what to go when SPC readings are not within the tolerance 
stayed the same improved Improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



Please complete the following sentences about classes you have taJcen 
1. I liked best about the class that 



2. The class helped me to 



3. 2 was disappointed that the class didn't address. 



4. In future I would like this class to 
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Name: Position; D epartmen t 

Department: 

Date: 



1. I currently have employees enrolled, in the Workplace Education Program 

2. To date I have had workers enrolled 

3. I do/do not acttrcfrr rucoiiragin woAciS to enroll in the program 

4. I do/do not understand the workplace literacy program In place now 

9. I do/do not keep track of the errors made and who made them when I check my datty 



e\ When employees go to clam 1 And that it Interrupts my production 

sometimes often 



7. I do/do not beneve that I abould Identify areas needing Impr o v e m ent of specific 
enrolled employees with the coordinator of the program 

8. Please circle which alternative yon behove that participation in the education 
program has proved for workers under your supervisi on 



•Workers are not afraid to ask questions about things they don't understand 
stayed the same improved improved a lot 

•Workers understand directions 

stayed the same improved improved a lot 

•Workers figure out ways do things themselves 

stayed the same improved improved a lot 

•Workers discover and apply new /better ways to do things at work 
stayed the same improved improved a lot 

•Workers work well with co-workers 

stayed the same improved tmprczied a lot 

•Workers are developing sense of team work and team identity 
stayed the same improved improved a lot 

•Workers can describe their jobs 

stayed the same improved Improved a lot 



"Workers can relate their Jobs to overall production of capacitors 

stayed the same improved 

"Workers inspect parts of their equipment 

stayed the same improved 

"Workers practice maintenance of their equipment 

stayed the same improved 

"Workers practice cleanliness of their equipment/surroundings 

stayed the same improved 

"Workers take safety precautions 

stayed the same improved 

"Workers understand need for Quality Control 

stayed the same improved 

"Workers understand the purpose of having an MPI 

stayed the same improved 

"Workers can follow their MPI 

stayed the same improved 

"Workers can follow MPIs from other sections 

stayed the same improved 

"Workers can read graphs around the plant 

stayed the same improved 

"Workers can construct a graph 

stayed the same improved 

"Workers can understand SPC procedures In the plant 

stayed the same improved 

"Workers could take part in SPC procedures If asked to do so 

stayed the some improved 

"Workers understand Right to Know 

stayed the same improved 



improved a lot 



improved a tot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 



improved a lot 
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9. 1 think the program needs improvement in the following ways: 



10. Any additional comments or ideas 



Thsnkyon 
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Hampden Paper Company 



CASE STUDY: 
OUTCOME STUDY AT HAMPDEN PAPER COMPANY 



Background: Hampden Paper Company, Holyoke 

Hampden Papers is a small (180 workers), family-owned paper manufacturing 
company situated along the Connecticut River. As recently as fifteen years ago, 
flint stones were still gathered along the river's banks, then bevelled and used 
as components of flint machines. Flint machines passed reams of paper around 
giant rollers as the bevelled stone treated the surface of the paper, imparting 
shine and texture. The flint machines at Hampden Paper produced about 1500 
lineal feet of paper per day- the same amount that their new computerized, 
mufti-function machine can now produce in a minute. This example highlights 
the technological changes which this company is undergoing and its need for 
workplace education. Limited-English-proficient (LEP) workers can no longer 
run the old machines with minimal instructions and cues from supervisors. As 
the work at Hampden Paper becomes more skilled and more customized, 
greater demands are made on workers to communicate, collaborate, problem 
solve, innovate, perform quality control, and perform more than just one job. 
Education is seen as a primary strategy for sustaining this massive change at 
Hampden Paper. The union at Hampden Paper is the Graphic 
Communications International Union, Local 4813. 

At present, all levels of ABE/GED and ESL are offered on an incfividualized, by- 
appointment basis in the learning center (classroom). Students average 2-5 
hours per week, and the teacher/coordinator is available for 25 hours per week. 
There are two 20-week cycles, and the learning center is open on a more 
limited basis in the summer. The classroom is on-site and used full-time as a 
learning center. Workers attend class on company time with full pay. 

In addition to the instruction currently offered, a "certification" program will soon 
be developed. This will mean that workers can become certified to perform as 
many operations in the production process as they choose to learn by enrolling 
in - and completing- job-certification programs. Their pay will rise accordingly. 
In this company, education is a strategy for transforming the methods of 
production and for retaining a loyal workforce. As a result, recruitment and 
regular class attendance are critical. 

From management's perspective, the "flexible" (that is, multi-skilled) workforce 
created through the certification program will enhance competitiveness and 
profitability by creating high-quality product in the least amount of time with the 
minimum amount of waste. From the worker's perspective, there is opportunity 
for mobility and pay increase, a more-creative and responsible relationship to 
the product being produced, and enhanced collaborative relationships with 
other workers. In addition, workers may realize benefits in relationships with 
family members and in other organizations or community/service groups. The 
union stands to establish a highly-skilled, well-organized workforce, which 



cannot be displaced by unskilled or lower-paid workers. This is workplace 
education defined as a classic win/win situation. 

Description of the evaluation process: 

The make-up of the evaluation team: The Hampden Paper evaluation team 
was composed of the president of the company; a supervisor; the vice president 
of the union, a student in the program, and two representatives of the education- 
provider agency, the International Language Institute (!U). One of these ILI 
representatives is the instructor/on-site coordinator for the program; the other 
serves as program coordinator. 

The process : The Hampden Paper team followed the ten-step process 
described in Section il of the report of the MWEI Cycle 4 Evaluation. AH 
members of the team were active participants in the process, although the 
student representative is limited-Engflsh-proficient and found it difficult to keep 
up with the pace and complexity of the English-language discussion. Most of 
the day-to-day work of designing and using the data-gathering instruments, and 
analyzing and presenting the data, was left in the hands of the program 
coordinator. 

What was te amed about the education program : See Section 11, Step 8 of the 
report of the MWEI Cycle 4 Evaluation.for a summary of the Hampden Paper 
team's findings, and the attached report by the program coordinator. 

What was teamed about the evaluation process: The teacher/on site 
coordinator was on maternity leave for both the January workshop and the first 
team meeting. This meant that she had to catch up quickly - and hopefully not 
be overwhelmed by the amount of work that fell to her - by the time the team 
was ready to meet for the second time. The teacher picked up the work without 
missing a step and was key to conducting the rest of the evaluation. The 
supervisor in the Hampden Paper team played a particularly active role at the 
beginning of the study. He substantially revised the draft supervisor report 
which the consultant had submitted for review, and helped the team to clarify 
the issues that were important for the supervisors- both in the questionnaire 
and in the data-gathering methods to be used. Unfortunately, this person came 
ill midway through the evaluation project and could not attend the final two team 
meetings. 

The "ripple" value of convening an evaluation team with representatives of all 
the key players was highlighted at Hampden Paper. The team meetings 
allowed these players to discuss two important issues with the company 
president that they would not have had the opportunity to discuss otherwise: 
the lack of Interest/support of some supervisors for the education program, and 
the long-term plans for a worker education/certification program tied to Job 
mobility and pay increase. 
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Attached data-gathering Instruments, summaries of findings, and related 
documents: 

Data-gathering instruments: 

- Questionnaire for ESL/ABE students 
~ Questionnaire for GED students 

Questionnaire for computer students 

• Each of these questionnaires contains questions on job specific 
skills, the ASTD/nigher order skills, and competencies matched to the 
class. 

- Questionnaire for Union Members 

- Questionnaire for Supervisors 

Summary of findings: 

- Outcome Study Final Report 
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Outcomes Study 
Hampden Papers, Inc. 
Final Report 
November 27, 1991 



INTRODUCTION AND OVERVIEW 

In June and July, 1991 a survey of workplace education 
students and their supervisors at Hampden Papers was 
conducted. Students were asked to rate their improvement in 
various areas of learning in the classroom, at work, at home, 
and in the community since enrolling in the Workplace 
Education Program. They were also asked about their union 
involvement as a result of participating in the progxam. 
Supervisors were asked about their attitude towards the 
classes, and the effect of the workplace program on 
production and worker skills. 

Student and supgrv sor responses were entered in the 
classroom computer, then c ross- tabulated by program 
(AJ£, GED, Computer ) and by department. First and second shift 
students were not separated. t \ 5 3K students account for 9 of 
the 20 students surveyed, computer students for 6, nnd GED 
students for 5. Students were given the option of circling 
"pre" on each question, indicating that they had a skill 
before entering the program. This was intended to 

distinguish between students already skilled in an area, and 
those who showed no improvement in that area. Percentages 
used discount any answers of "pre" to ensure a clear 
representation of improvement. 



FINDINGS: JOB-SPECIFIC SKILLS 

General Math Ski lis. 40. J of tne responding supervisors 
note some improvement. 83". of A BE students note 

improvement (50% "a lot" and 33" "a little") in basic 
math, G5% in usin- percentages (42.3'J "a lot" and 42.8". "a 
little"), and 71".; in usin^, fractions (42. 8.n " □ lot" and 
23. 6* a little"). 75- of GED students note improvement 
in using fractions, decimals and pprccnts (25% "a lot" and 
50Z " a little"). 

2. Measurement Skills. Only 1 of the responding supervisors, 
or lo.Gr,, indicated an improvement in the measuring skills 
shown by students. Conversely, 100* of the ABE students 
(7 out of 7) not having the skill previously indicated an 
improvement in this area (42.9:; "a lot" nnd 57Z "a 
little"). None of the -5 responding students said they had 
the skill previously. »h:j of CE'J students note 
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improvement, 33% " a lot" and 33% "a little". Supervisors 
were asked about reading measuring devices accurately 
while students were asked about using measurements, 
leaving room for a discrepancy in results. 

3. Readinp, Production Tickets . 60% of the responding 
; supervisors note some improvement in accuracy. 87.5% of 

ABE students note improvement f with only 1 person 
indicating "pre". 62.52 note a lot of improvement. GED 
and computer students were not asked this question. 

4. Writing Accurate Reports. 28^ of all supervisors note 
some improvement. 66. GZ of ABE students note improvement, 
44.4% "a lot". lione of the students indicated "pre". 
This and reading production tickets are clearly skills 
that students believe they need work on, despite the fact 
that everyone has said during conferences with the teacher 
that they have no trouble with production tickets. The 
word "accurate" is ambiguous, leaving room for a 
discrepancy in answers. Computer and GED students were 
not asked this question. 

5. Understanding directions, 42. 9/* of supervisors note sone 
improvement. 71.4:; of AHF enrollees indicated a lot of 
improvement. Only 1 Cr.H and 1 computer student note any 
improvement, the others shoving a prior skill. It is to 
be expected that CiEi) and computer students will have 
better communication skills than AHE students, a group 
which includes r.SL students. 

Communicating Needs to Supervisor. 50% of responding 
supervisors note some improvement, with 1 , or IG.GZl 
noting a lot of i nprovenent . Thi s is the only time a 
supervisor answored "a lot' 1 in this survey, 85.7% of ail 
students note improvement, 2S.5 c ; "a lot" and 66. 6% "a 
little". ASH students note no prior skill, whereas 40" of 
GEi) students and 06.6/, of conputer students indicated n 
prior skill. This is understandable as the self- 

confidence of people in these groups is likely to be 
hi :;her . 

7. Asking Questions Clearly . 40.", of the supervisors note 
improvement , as did 7 571 of all students without a prior 
skill. The biy'?;est ircprovenent was in the A3E group, 

8. W o r k i n g Wei 1 * i t h Ot hers. 2S f 5"! of supervisors note sone 
improvement. o3.3, of nil students report improvement, 
6o.6% noting a lot. 

Solving Problems Independently. 1 supervisor note 

innrovement (16.6 i), while 92. 9;I of all students said they 
improved (57. i;; "h lot ") . The student question differed 
fron the supervisor question: f, I don't wait to he told 
what to do ,T . K> 
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10. Of f er i np» Opinions and Ideas . 28.6% of supervisors and 
85. 7% of all students indicated improvement* Fewer 
students said they improved a lot, only 33.3% as compared 
to stronger improvement note previously. "Pre" responses 
were low, but increased with skill level. This would seem 

. to be an area to focus on with students in all components 
of the program. 

11. Understanding Role in Hampden Papers. 16.6% of responding 
supervisors indicated there was an improvement here. 
78.6% of all students note improvement f 57.1% note a lot. 
80% of students in the Fancy Department and 83% of those 
in the Secei vi ng/Power Wind Department improved a lot. 

OBSERVATIONS 

The Receiving/Power Wind Department note a large improvement 
in the way workers communicate their needs on the job, and 
tiie most overall improvement of all departments. 

The tendency of supervisors to report less improvement in 
skills surveyed nay be explained by the fact that supervisors 
do not see discrete learning, rather overall competency* A 
student i:\ny have learned many of the steps involved in 
working with fractions, but may not yet be able to do the 
multiplication required for a certain job. The student 
would, therefore, see much more progress in fractions than 
would the supervisor. In addition, what students are doin^ 
in their class work may not always be directly related to the 
job they are doing at the time. Finally, the supervisors 
were not ;;iven the option of answering "pre", or "not 
applicable 11 for their questions. This may have skewed the 
results. 

Findings of Production-related Supervisor Questions 

-lOi)Z of the supervisors said they actively encourage 
enrollment in the workplace program, and that they understand 
the progran in place at the time of the survey, 
-iio.o keep track of errors made and who made then. 
-The receiving/Power Wind supervisors said their production 
was often disrupted by students coming to class. This 
follows from the fact that this department has the highest 
enrollment (9 out of 20). 



of the supervisors think they should identify areas 
needing improvement in their workers with the teacher. 
-Areas of t he program cited as needing improvement were: 

.lore job-specific work in the classroom. 

rind a way to keep students attending class. 
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-The supervisors said they had a total of 15 workers who they 
feel should enroll in the program, but couldn't at the time. 
-One supervisor cited one of his workers as having notably 
improved reading skills and self-esteem. 



FINDINGS: ASTD CATEGORIES (Questions not covered above.) 

Like to learn. 85.7% of students without prior skills 
note improvement, 57.12 a lot and 28.6% a little. 100% of 
GED students without prior skill report improvement. Only 
1 marked "pre". These students have clear short-term 
goals, and daily progress can be measured with the 
competency-based curriculum they follow. 

2. Know where to_ get information. 85.7% of students without 
prior skill note improvement, 35.7% a lot and 50% a 
little. This is to be expected with the independent 
learning environment which has been created in the 
classroom. 

3. Listen to my co-workers . GED and computer students show a 
prior skill. 75% of ABE students (there was only 1 "pre") 
note improvement. 



4. Listen to family and 
students show a prior 
1 "pre") note 



friends. 



skill, 
improvement, 62.5% 



Again, GED and computer 
87.5% of ABE students (only 
a lot . 



5. Think of 



better ways to do thinp.s at work. 66.6% of all 

students without this skill note a lot of improvement, and 

little. A?«ain, ABE students report the 



20% note 
greatest 



a 



improvement 



6 * Tr y better ways to do thinp,s at work. 92.3 of all 
students note improvement, 61.5% a lot. This again can be 
understood considering the independence fostered in the 



c lassr oom. 
initiative. 



This has an effect 



on 



self-esteem and 



Value opinions and contributions. 
without prior skill note improvement, 
of ABE and computer students without 
improvement . 



91. 6& of students 
41.62 a lot. 100% 
prior skill report 
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3 . Support others 
all students 

ff of only 



pre 
i improvement 
Wind Department 
lot of improvement, 
largest enrollment, 



participation in the program. 88.8% of 
note improvement, 77.7% a lot. There was a 
2 f making this the largest reported 
in the survey. 87.5% of the Receiving/Power 
and 65.7% of the Fancy Department note a 
These two departnen ts have the 
and perhaps the best comraderie. It 



is interesting to see that GED 
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students were 



the most enthusiastic in their responses. No one answered 
"stayed the same", only 1 answered "improved a little", 
and there was only 1 "pre" in this group. As explained 
above, these two groups have more "easily" attainable 
short-term goals, which may generate more enthusiasm. 

Support family and friends ' efforts to learn . 81.3% of 
all students without prior skill report improvement, 75% a 
lot. 100% of GED students answer "a lot". 

10. Understand long-term goals of Hampden Papers . 87. 5% of 
all students note improvement, 56.3% a lot. 100% of the 
Shipping/Embossing Department, 100% of the Fancy 
Department (only 1 "pre"), and 87.5% of the 
Receiving/Power Wind Department report improvement. 

11 . Component-specific competencies. Students report overall 
improvement in all areas surveyed, math, reading and 
vriting skills. ABE students note the biggest gain in 
most areas, and computer students report the least. 
Computer students also indicated "pre" most often in all 
areas. This is understandable considering the differing 
skill levels of each n r oup« 

-At least 85% of ABE students report improvement in all 
but 2 areas of reading, writing and math. 

-None of the ABE students and none of the GED students in 
4 out of 5 skills indicated "pre" in writing skills. All 
need work in this area. 

-GED students report overall improvement except in social 
studies and science areas, where 3 out of A had a prior 
skill. -There was less improvement in GED math skills, 
especially in more advanced areas. 

— 75% to 80% of ABE and GED students report improvement in 
solving word problems, reading tables and graphs, and in 
using measurement. 

-83% of computer students indicated improvement in 
applying programs and skills for job and home purposes. 



FINDINGS: UNION MEMBER SURVEY 

1. Awareness of union benefits. 55 % note an increase. This 
increase was the same across departments, and slightly 
higher in the Color/Coati np, Department. 

2. Desire to negotiate benefits on union contracts. 50,"! had 
an increased interest. This included 100% of 
Shipping/Embossing, 57% of the Fancy Department, and 66^ 
of computer students. This would perhaps be higher at the 
end of a contract period, when workers are ant ic i pa t i n f» 
new negotiations. 
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3» Interest in becoming a union officer. 10% report an 

increase {in the Receiving/Power Wind Department only), 
and 20% a decrease. This decrease may happen when workers 
enroll in the program as their time constraints and 
understanding of the complexities of union involvement 
increase. 

4. Understandinn union role in Workplace Education Program. 
65% report an increase. 100f of The Fancy Department 
enrollees note an increase. 



SUGGESTIONS FOR FUTURE SURVEYS 

-The wording of the questions needs to be the same for both 
supervisors and students to allow for more reliable cross- 
tabulation of responses. 

-Supervisors need to have the option of answering "pre" or 
"NA", as did the students. 

-Questions regarding job-specific skills need to be asked of 
all students, regardless of program enrolment. It nay be 
possible to find an correlation between simple involvement in 
the workplace education program and improved skills on the 
job. 

-Company and union representatives on the evaluation tenn 
suggested that the supervisors answer survey questions fur 
each of their workers to get a clearer picture of the 
program's effect on the shop floor. Information can also be 
used to improve the curriculum. 

-When the company has the capacity to measure senp, the 
"pre" and "post" figures can uo examined as a uospiulc 
indicator of program effectiveness. 

-It may also be helpful for supervisors to keep copies of 
employee reports from the beginning of their involvement in 
the program for comparison with later work, as additional 
evidence of program effectiveness. 

-A more sophisticated statistical computer program will ho 
used to analyze survey results in the future. This year's 
data may be retabulated using the new program. 

The questionnaires will be rewritten with the above in ain.l 
and distributed in June to supervisors, current students ami 
union members. As responses are tracked over time, results 
and insights will become more meaningful. The information 
gathered is expected to help in curriculum modification, .is 
well as in judging overall program effectiveness for funlcr, 
company management, and recruitment purposes. 
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CASE STUDY: 
OUTCOME STUDY AT HAMPDEN PAPER COMPANY 



Background: Hampton Paper Company, Holyoko 

Hampden Papers is a small (180 workers), family owned paper manufacturing 
company situated along the Connecticut River. As recently as fifteen years ago, 
flint stones were still harvested along the river's banks, then bevelled and used 
as components of fflnt machines. Flint machines passed reams of paper around 
giant rollers as the bevelled stone treated the surface of the paper, imparting 
shine and texture. The flint machines at Hampden Paper produced about 1500 
lineal feet of paper per day- the same amount that their new computerized, 
multi-function machine can now produce in a minute . This example highlights 
the technological changes which this company is undergoing and its need for 
workplace education. Limited English proficient (LEP) workers can no longer 
run the old machines with minimal instructions and cues from supervisors. As 
the work at Hampden Paper becomes more skilled and more customized, 
greater demands are made on workers to communicate, collaborate, problem 
solve, innovate, perform quality control, and perform more than just one job. 
Educations seen as a primary strategy for sustaining this massive change at 
Hampden Paper. The union at Hampden Pare is the 

At the present time, ail levels of ABE/GED and ESL are offered on an 
individualized, by appointment basis in the learning center (classroom). 
Students average 2-5 hours per week, and the teacher coordinator is available 
for 25 hours per week. There are two 20 week cycles, and the learning center is 
open on a more limited basis in the summer. The classroom is on-site and used 
full-time as a learning center. Workers attend class on company time with full 
pay. 

In addition to the instruction currently offered, a "certification" program will soon 
be developed. This will mean that workers can become certified to perform as 
many operations in the production process as they choose to learn through 
enrolling in - and completing- job certification programs. Their pay will rise 
accordingly. In this company, education will be a strategy for transforming the 
methods of production and for retaining a royal workforce. As a result, 
recruitment into and regular attends /ice in classes critical. 

From management's perspective, the "flexible" (i.e. multi-skilled) workforce 
created through the certification program will enhance competitiveness and 
profitability by creating high quality product in the least amount of time with the 
minimum amount of waste. From the worker's perspective, there is opportunity 
for mobility and pay increase, a more creative and responsible relationship to 
the product being produced, and enhanced, collaborative relationships with 
other workers. In addition, workers may realize benefits in relationships with 
family members and in other organizations or community/service groups. The 
union stands to establish a highly skilled, well organized workforce, which 
cannot be displaced by unskilled or lower paid workers. This is workplace 
education defined as a classic win/win situation 
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Description of the evaluation process: 



The make-up of the evaluation team; The Hansen Paper evaluation team 
was composed of the President of the company; a supervisor; the Vice 
President of the union, a student in the program, and two representatives of the 
educatio^provkler agency, the I'/rternational Language Institute. One of these 
representatives of the International Language Institute is the instructor/on site 
coordinator for the program; the other serves as program coordinator. 

The process: The Hampden Paper team followed the evaluation process 
described in Section II of this report. All members of the team were active 
participants in the process, although the student representative is limited- 
English proficient and found It difficult to keep up with the pace and complexity 
of the English language cflscussion. Most of the day-to-day work of designing 
and using the dam-gathering instruments, and analyzing and presenting the 
data, was left in the hands of the program coordinator and his fellow instructor. 

What was learned flho.it tha education pmpmnv See Section II, Step 7 Of this 

report for summary of the Hampden Paper team's findings. 

What was learned about the evaluation pjagess; The teacher/on site 
coordinator was on maternity leave for both the January workshop and the first 
team meeting. This meant that she had to catch up qutekiy and hopefully not be 
overwhelmed by the amount of work that fell to her, by the time the team was 
ready to meet for the second time. The teacher picked up the work without 
missing a step and was key to conducting the rest of the evaluation. The 
supervisor in the Hampden paper team played a particularly active role. He 
substantially revised the draft supervisor report which the consultant had 
submitted for review, and helped the team to clarity the issues that were 
important for the supervisors- both in the questionnaire and in the dam 
gathering methods to be used. The "rippie" value of convening an evaluation 
team with representatives of all the key players was highlighted at Hampden 
Paper. The team meetings allowed these players to discuss two important 
issues with the company President that they would not have had the opportunity 
to discuss otherwise: the lack of Interest/support of some supervisors in the 
worWace education program, and the long term plans for a worker 
education/certification program tied to job mobility and pay increase. 

Data-gathering instruments and related documents: 

Data-gathering instruments: 

~ Questionnaire for ESL/ABE students 

- Questionnaire for GED students 

- Questionnaire for computer students 

• Each of these questionnaires contains questions on job specific 
skills, the ASTD/higher order skills, and competencies matched to the 
class. 

- Questionnaire for Union Members 

- Questionnaire for Supervisors 



Summary of findings: 

- Outcome Study Preliminary Oral Report, September 18, 1991 
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Observations Checklist for supervisors 



Please answer the following questions for the workers you 
have who axe or have been enrolled in the Workplace Education 
Program. You do not need to put your name on this form. 



X. I currently have employees enrolled in the Workplace 

Education Program. 

2. I do/do not actively encourage workers to enroll in the 
program. 

3. I do/do not understand the workplace education program in 
place now. 

4. I do/do not keep track of the errors made and who made 
them when I check my daily production reports. 

5. when employees go to class I find that it interrupts my 
production. 

never seldom sometimes often 

6. I do/do not believe that I should identify areas needing 
improvement of specific enrolled employees with the 
teacher of the program. 

7. Please indicate if you believe that participation in the 
Workplace Education Program has improved for workers under 
your supervision. 

a. math skills 

stayed the same improved improved a lot 

b. read production tickets accurately 

stayed the same improved improved a lot 

c. read measuring devices accurately 

stayed the same improved 

d. write accurate reports 

stayed the same improved improved a lot 

e. understand directions 

stayed the same improved improved a lot 

f . communicate needs to you 

stayed the same improved improved a lot 

g. ask questions clearly/without hesitation 

stayed the same improved improved a lot 
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h. need less translation/clarification 

stayed the sane improved improved a lot 

i. work well with others 

stayed the same improved improved a lot 

j. solve problems independently 

stayed the same improved improved a lot 

k. offer opinions and ideas 

stayed the same improved improved a lot 

1. understand role in Hampden Papers 

stayed the same improved improved a lot 



8. I think the program needs improvement in the following 
areas: 



9. Is there anything specific you want to say about a 
particular worker? 



10. I have approximately __ employees in my department I 
feel should enroll In the program, but at this time are 
either unable or unwilling to do so. 



11. Any additional comments or ideas: 
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Checklist for Union Members/Stewards 



Do you think that participation in your workplace education program 
has changed the following for union members? Please circle 
decreased, increased or stayed the same after each item. 

1. Awareness of range of benefits offered through the union 
decreased increased stayed the same 

Z Identifying problems on shop floor as union concerns 
decreased increased stayed the same 

3. Understanding the grievance procedure 

decreased increased stayed the same 

4. Interest in becoming a steward or union officer 

deceased increased stayed the same 

5. Desire to negotiate benefits on union contracts 

decreased increased stayed the same 

6. Understanding the union role in creating and supporting your 
workplace education program 

decreased increased stayed the same 

7. Understanding the terms of your union contract (as reflected in 
the curriculum or from other sources) 

decreased increased stayed the same 

8. Anything else that has changed? Please specify 

decreased increased stayed the same 

255 



5/28 

Self -Report Checklist 
Workplace Education Program 
GEO Students 

Do you think that participation in your workplace education 
program has changed the following for you? Please oirole ones 
pre-you had the skill before entering the program 

1- stayed the same 

2- increased a little 

3- increased a lot 

1. Foundation - knowing how to learn 

a. I like to learn. 

pre 12 3 

b. I'm not afraid to ask questions about something I don't know. 

pre 12 3 

c. I know where to go for information to answer my questions (to 
a book, a person, etc.). 

pre 12 3 

2* Competenoe - reading, writing and computation 
READING SKILLS 

a. I can understand the main idea and supporting details. 

pre 123 

b. I can make inferences from readings (read between the lines) . 

pre 12 3 

c. I can analyze and apply new ideas from a reading. 

pre 12 3 

d. I can understand social studies readings. 

pre 123 

e. I can understand science readings. 

pre 123 

d. I can understand poetry, drama, fiction and non-fiction, 
pre 12 3 

WRITING SKILLS 

a. I can understand and use good sentence structure. 

pre 12 3 

b. I can understand and use good grammar. 

pre 12 3 
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c. I can use correct spelling, punctuation and capitalization. 

pre 12 3 

d. I can write a clear and well-organized paragraph. 

pre 12 3 

e. I can write a clear and well-organized essay. 

pre 12 3 



MATH 
a. I 


SKILLS 

can use fractions, decimals and percents. 
pre 12 3 


b. 


i 


can 


use ratios, proportions and 
pre i 2 


probability. 
3 


c. 


X 


can 


solve geometry problems, 
pre i 2 


r, 

4 


d. 


I 


can 


solve algebra problems, 
pre i 2 


3 


e. 


I 


can 


solve word problems, 
pre i 2 


3 


f. 


I 


can 


read tables and graphs, 
pre l 2 


3 


g- 


I 


can 


use liquid, dry and linear measurement, 
pre 12 3 


h. 


I 


can 


use exponents, roots and scientific notation, 
pre 12 3 



3. Communication - listening and speaking 
a. I can understand directions. 

pre 12 3 



b. I listen to my co-workers. 

pre 12 3 

c. I listen to my family members and friends. 

pre 12 3 

d. I can ask questions clearly. 

pre 12 3 

for ESL students 

e. I need less repeating or translation. 

pre 12 3 

4. Adaptability - creative thinking and problem solving 
a. I don't wait for someone else to tell me what to do. 
pre 12 3 



257 



b. I think about new or better ways to do things at work. 

pre 12 3 

c. I try new or better ways to do things at work. 

pre 12 3 

5- Personal Management - self-esteem, goal setting, motivation, 

personal and career development 
a. I value my opinions and contributions. 



b. I offer my opinions to others. 

pre 12 3 

c. I plan ahead in my personal life. 

pre 12 3 

d. I plan ahead at work. 

pre 12 3 

e. I am comfortable working alone. 

pre 12 3 

f . I am a more active member of my community (church, union, 
community center) . 

pre l 2 3 

6. Group Effectiveness - interpersonal skills, negotiation and 
teamwork 

a. I work well with members of my work station. 

pre 123 

b. I am willing to talk about my needs and the needs of my group 
with my supervisor. 

pre 12 3 

c. I support others 1 efforts to participate in the workplace 
education program. 

pre 12 3 

d. I support family and friends 1 efforts to learn. 

pre 12 3 

7. Influence 

a. I understand the relationship between the parts of the work 
process I do and the whole paper production process. 

pre 12 3 

b. I understand the long term goals of Hampden Papers, Inc. and 
the necessary role that education plays in reaching those 
goals. 

pre 12 3 



258 



5/28 



Self -Report Checklist 
Workplace Education Program 
Computer students 



Do you think that participation in your workplace education 
program has changed the following for you? Pleas* circle onss 

prs-you had the skill before entering the program 

1- stayed the same 

2 - increased a little 

3- increased a lot 

1. Foundation - knowing how to learn 

a. I like to learn. 

pre 12 3 

b. I'm not afraid to ask questions about something I don't know. 

pre 12 3 

c. I know where to go for information to answer my questions (to 
a book, a person, etc.). 

pre 12 3 

2. competence 

a. 1 can use Professional Write for writing and editing. 

pre l 2 3 

b. I can use Professional Write for forms and page design. 

pre i 2 3 

c. I can use Professional File to make simple reports. 

pre 12 3 

d. I can use Professional File to make calculations. 

Pre 1 2 3 

e. I can use these programs for practical job or home purposes. 

pre 1 2 3 

3. Communication - listening and speaking 

a. I can understand directions. 

pre 12 3 

b. I listen to my co-workers. 

pre 12 3 

c. I listen to my family members and friends. 

pre 12 3 



d. I can ask questions clearly. 

pre 12 3 

for ESL students 

e. I need less repeating or translation. 

pre 12 3 

4. adaptability - creative thinking and problem solving 

a. I don't wait for someone else to tell me what to do. 

pre 12 3 

b. I think about new or better ways to do things at work. 

pre 12 3 

c. I try new or better ways to do things at work. 

pre 12 3 

5. Personal Management - self-esteem, goal setting, motivation, 
personal and career development 

a. I value my opinions and contributions. 

pre 12 3 

b. I offer my opirions to others. 

pre 12 3 

c. I plan ahead in my personal life. 

pre 12 3 

d. I plan ahead at work. 

pre 12 3 

e. I am comfortable working alone. 

pre 12 3 

f. I am a more active member of my community (church, union, 
community center) . 

pre 12 3 

6* Group Effectiveness - interpersonal skills, negotiation and 
teamwork 

a. I work well with members of my work station. 

pre 12 3 

b. I am willing to talk about my needs and the needs of my group 
with my supervisor. 

pre 12 3 

c. I support others* efforts to participate in the workplace 
education program. 

pre 12 3 
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d. I support family and friends' efforts to learn, 
pre 12 3 



7. influence 

a. I understand the relationship between the parts of the work 
process I do and the whole paper production process. 

pre 12 3 

b. I understand the long term goals of Hampden Papers, Inc. and 
the necessary role that education plays in reaching those 
goals. 

pre 12 3 
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Self -Report Checklist 
Workplace Education Program 
ABE Students 

Do you think that participation in your workplace education 
program has changed the following for you? Please eirele ones 

pre-you had the skill before entering the program 

1- atayed the same 

2 - increased a little 

3 - increased a lot 

1. Foundation - knowing how to learn 

a. I like to learn. 

pre 12 3 

b. I'm not afraid to ask questions about something I don't know. 

pre 12 3 

c. I know where to go for information to answer ny questions (to 
a book, a person, etc.)* 

pre 12 3 

2. Competence - reading, writing and computation 
READING SKILLS 

a. I can read a production ticket. 

pre 12 3 

b. I can read graphs, charts, grids and maps. 

pre 12 3 

c. I can understand the main idea and details of what I am 
reading. 

pre 12 3 

d. I can make inferences from what I am reading (read between the 
lines) . 

pre 12 3 

e. I have built up my vocabulary. 

pre 12 3 

WRITING SKILLS 

a. I can complete job tickets and reports. 

pre 12 3 

b. I can use the word processor. 

pre 12 3 
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c. I can express my ideas clearly in written form (words, 
sentences, paragraphs, letters, essays) specify which 



pr# I 2 3 

MATH SKILLS 

a. I can do basic addition, subtraction, multiplication, and 
division. 

pre 12 3 

b. I can figure simple percentages. 

pre 12 3 

c. I can add and subtract simple fractions. 

pre 12 3 

d. I can use measurements. 

pre 12 3 

3* Communication - listening and speaking 

a. I can understand directions. 

pre 12 3 

b. I listen to my co-workers. 

pre 12 3 

c. I listen to my family members and friends. 

pre 12 3 

d. I can ask questions clearly. 

pre 12 3 

for ESL students 

e. I need less repeating or translation. 

pre 12 3 

4. adaptability - creative thinking and problem solving 

a. I don't wait for someone else to tell me what to do. 

pre 12 3 

b. I think about new or better ways to do things at work. 

pre 123 

c. I try new or better ways to do things at work. 

pre 12 3 
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GRAND TOTAL 

821950 

5. Personal Management - self-esteem, goal setting, motivation, 
personal and career development 

a. I value my opinions and contributions. 

pre 123 

b. I offer my opinions to others. 

pre 12 3 

c. I plan ahead in my personal life. 

pre 12 3 

d. I plan ahead at work. 

pre 12 3 

e. I am comfortable working alone. 

pre 12 3 

f . I am a more active member of my community (church, union, 
community center) . 

pre 12 3 

6. Group effectiveness - interpersonal skills, negotiation and 
teamwork 

a. I work well with members of my work station. 

pre 12 3 

b. I am willing to talk about my needs and the needs of my group 
with my supervisor. 

pre 12 3 

c. I support others 1 efforts to participate in the workplace 
education program. 

pre 12 3 

d. I support family and friends' efforts to learn. 

pre 12 3 

7. influence 

a. I understand the relationship between the parts of the work 
process I do and the whole paper production process. 

pre 12 3 

b. I understand the long term goals of Hampden Papers, Inc. and 
the necessary role that education plays in reaching those 
goals. 

pre 12 3 
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South Cove Manor Nursing Home 
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CASE STUDY: 
OUTCOME STUDY AT SOUTH COVE MANOR 



Background: South Cove Manor Nursing Home, Boston 

The "product" which staff produce in this one-of-a- kind nursing home for elderly 
Chinese people in the Commonwealth is quality patient care. Differences in 
culture and language between mainly Chinese entry-level workers and nurses' 
aides and native-English-speaking supervisory staff can compromise quality of 
patient care in serious ways. For example, the seemingly small error of placing 
a salted or sugared food item on a food tray for a patient with restricted dietary 
needs may cause health problems. The on-site ESL program facilitates 
communication between English-speaking licensed staff (R.N.'s and M.D.'s) 
and Chinese-speaking nurses aloes and other entry-level workers, thereby 
improving the quality of patient care. 

The South Cove Manor Nursing Home (SCM) is a private, non-profit nursing 
home located on the outskirts of Boston's Chinatown. Members of the Chinese- 
American community worked for over ten years to make the dream of a long- 
term-care facility for elderly Chinese a reality and take pride in the fact that it is 
one of only three nursing homes in the nation oriented to the social, dietary, and 
medical needs of the elderly Chinese community. Approximately 70 percent of 
the residents are Chinese. As the general population ages, nursing homes 
increase in number, requiring more licensed and unlicensed staff to work in 
them. At the same time, the nation and the State of Massachusetts are facing a 
shortage of nursing personnel; and the demographics of people likely to find 
employment in entry-level nursing home jobs are also changing. They are 
likely to be recent immigrants to the United States and limited-English- 
proficient. 

Like other nursing homes throughout the State of Massachusetts, South Cove 
Manor faces problems with recruitment of licensed staff-- especially R.N.'s- 
and with retention of unlicensed staff. But these problems are compounded by 
the bilingual needs of the agency, in order to provide quality care to its 
residents, the agency needs nurses who are bilingual in Chinese and English. 
They are in especially short supply. The agency also needs unlicensed staff to 
work in the dietary and laundry departments and to work as nurses' aids, who 
can speak English well enough to both take directions in English from English- 
speaking supervisory staff and interact with English-speaking residents. Entry 
level workers are primarily but not exclusively Chinese. Some are Haitian and 
speak French. 

An on-site ESL program addresses these problems in several ways. It: 

1 . ) Creates an environment of acceptance of English as a common language 

within the bicuitural setting. 

2. ) Creates a work environment where education Is valued and made available 

in the service of quality patient care. 
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3. ) Helps nurses' aides who are Chinese speaking work toward becoming 

licensed as LP.N.'s and RN.'s, thereby increasing the pool of licensed, 
bilingual nurses in the long term. 

4. ) Increases the written and oral English language skills of unlicensed staff, 

thereby reducing dependence on translators and interpreters. 

5. ) Creates an opportunity for underemployed staff to develop English language 

skills and move up the career ladder at South Cove Manor if possible, or 
elsewhere. 



The South Cove Manor ESL program offers ESL instruction a the beginner and 
intermediate levels. There are 4 beginner classes averaging nine students 
each, which meet six hours a week; and an intermediate class with nine 
students, which meets two hours a week. There are two 22 week instructional 
cycles per year. Classes are taught in the conference room near the front 
entrance on the first floor, and in the physical therapy room. They are taught by 
the on-site coorolnator/lnstructor and instructor. In a addition to classes, an 
average of ten workshops are offered per year, many focur : ~tg on career 
development. Seven students also chose to participate in a ©elf instructional 
VESL program, and 15 students are partnered with volunteer "buddies" who 
assist them with language learning. The ESL program does not provide 
release time. 

Description of the evaluation process: 

The make-up of the evaluation taamr The SCM evaluation team was composed 
of the head administrator of the nursing home; two high-level representatives of 
the management company which operate the home; a nurse/supervisor; and 
two representatives of the education-provider agency, the Chinese American 
Civic Association. (The CACA representatives were both instructors in the 
program and one of them also served as education program coordinator.) Early 
in the process, the instructors also tried to include students as active 
participants in the team. However, this was not encouraged by the SCM 
management on the grounds that it took students away from their jobs. 

The process: The SCM team followed the ten-step process outlined in Section 
II of the report of the MWEI Cycle 4 Evaluation. Most of the day-to-day work of 
designing and using the data-gathering instruments, and analyzing and 
presenting the data, was left in the hands of the program coordinator and his 
fellow Instructor. 

What was learned about the education program: See Section II, Step 8 of the 
report of the MWEI Cycle 4 Evaluation.for a summary of the SCM team's 
findings, and the attached report by the program coordinator. 

What was teamed ahout the evaluation prnnasg- The SCM team - led by the 
coordinator and his fallow instructor - was outstanding in the way it clarified the 
outcomes it wanted to focus on, designed and used its data-gathering 
instruments, and then analyzed the resulting data. However, the team's effort 
was limited by its tendency to place most of the responsibility for the process on 
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the shoulders of the education providers. Despite the high quality of the 
evaluation work being done, as the project drew to a close it appeared that 
management had tost interest in participating in the study and using the 
information produced. This was puzzling because SCM had initially expressed 
clear interest in producing "quantitative data" about the impact of the program 
on the quality of patient care. Although it was not clear what SCM might do with 
the work done in the project, it was dear that the education provider - the 
Chinese American Civic Association - intended to uake active use of the 
experience. CACA planned to implement a similar outcome study process in 
the other education programs it operates in Boston's Chinese community. 

Attached data-gathering instruments, summaries of findings, and related 
documents: 

Data-gathering instruments: 

- Simulations 

- Flow Sheet Exercise 

- Questionnaire for Supervisory and Licensed Staff 

- Resident Interview 

- Questionnaire for Nursi, .^ Assistants 

Summary of findings: 

- Outcome Study Preliminary Report 
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Has SCM changed the way it operates in order to most 
effectively make use of the HAs* stills ? 

• According to HAs, 50% of the English speaking supervisory and licensed staff pay more 

attention to their reports . 
- HAs' reports haw enabled the DON in 66.6 9&of all cases to adjust the patient care plan 

as necessary. 

« theEnglish speaking licensed staff in 50% 



the bilingual supervisors in 91 .6 96 



rts*o the DON, patient car 



• In 25 95 of all reports^) the DON, patient care adjustments have led to improvements in 

the resident's condition. 

• In 66 .6 % the English speaking nurses, 



• In 1 00 96 the bilingual supervisors , . 

Si 

Has SCM become less dependent on interpreters and 
translations ? 

• 66.6 96 of the NAs feel that English speaking staff don't need an interpreter (anymore) to 

communicate vith them. 

• The DON has become less depenedjnt on interpreting vhen interacting vith 41 .6 % of 

the NAs. 

• English speaking nurses feel they have become less dependent on interpreting in 66.6 % 

Of thff C8SCS* 

• The DON feels she does not need an interpreter in 50 % of all cases. 

• English speaking nurses say they don't need an interpreter in 66.6 9& of all cases. 

■ Bflmgnal supervisors say they interpret less for NAs in 91. 6 95 of all cases and that 
they depend less on interpreting to give instruction in 53.3 9& of all cases. 

Have NAs benefitted from the program in ways not directly 
related to their job ? 

• The general proficiency of ALL participants has developed considerably. 
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Outcome Study 
South Cove Manor Workplace Education Program 

July 25, 1991 

The Questions 



Learning of Eagfidpaate 

Do participants tetter master the skills that are critical to the delivery of 
patient care of high quality ? 

Job Perform a nce or Transfer of Leaning to tfte lob 

Has the number of positive resident/nursing assistant interactions 
increased ? Has the number of misunderstandings between residents and 
nursing assistants decreased ? 

Do nursing assistants properly use English to report changes in the 
condition of residents ? 

Do nursing assistants properly read and f iU out work-related documents in 
English ? 

QtgattfaatifinaJ p »rf W m *™* of Changs in the fiefravlOf Pf tfte 
Larger Organization 

Has the organization (i.e. South Cove Manor) changed (eg. quality of care) ? 

Has the organization itself changed the way it operates in order to most 
effectively make use of nursing assistants* skills ? 

Personal Developmen t of Nursing Assistants 

Have nursing assistants benefitted from the ESOL program in ways not 
directly related to their jobs ? 
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The Instruments 



• a series of classroom -based simulations of Job tasks that 
require the use of those English skills that are critical to the 
quality of patient care 

• a classroom-based flow sheet exercise that requires 
participants to read an English only flow sheet and to enter 
information on completion of tasks 

• an interview with, residents 

% 

• a survey given to English speaking licensed, supervisory, 
and training staff 

• a survey given to participants 



• a standardized language proficiency test (i.e. BEST-test) 
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Outcome Study 
South Cove Manor Workplace Education Program 

July 25, 199! 



Participant Profile 

The evaluation team decided to focus on those participants who had been 
in the program the longest All participants in the study were nursing 
assistants. 



Gender 



Age 



Male 1 
Female H 



36 1 

37 1 

38 1 

43 1 

46 1 

50 1 

51 2 

52 2 

53 1 
61 1 



Mean Age- 47.5 
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Ethnic Background and Gender 



Male Female 

Asian 1 10 

Haitian 0 1 

Mother Tongue 

Cantonese 1 1 

French Creole •* 1 

Educational Level 

Level 1 2 

Level 2 3 

Level 3 7 



1* elementary or less 

2* high school or equivalent in vocational training 
3- higher education 



7-3 6 
3-11 4 
11-7 0 
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Floor 



2 floor 

3 r<l floor 



7 
5 



Hours of Instruction 



64 

146 

216 
236 

255 
266 

316 
372 
392 

440 
456 



4 



Average - 292.41 
Country and Setting of Origin 



China (main land) 
Hong Kong 
Haiti 



10 
1 
1 



Rural 0 
Metropolitan 6 
Suburban 4 
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Length of Employment 



- 1 year 0 

1- 2 yrs. J 

2- 3 yrs. 5 

3- 4 yrs. 1 

4- 5 yrs. 0 
♦ 5 yrs. 3 



Previous or concurrent ESOL learning experience 

Yes 2 
No 10 

BilinguaUsm ' 

Cantonese /Mandarin 9 
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Resident Profiles 



Residents were selected by the case-mix nurse on the evaluation team. 
Their mental condition and their willingness to participate were taken into 
account Seven residents provided information about the nursing 
assistants in the outcome study. All of them interacted frequently with 
participants and had been at the nursing home for a while. 



Gender 



Male 
Female 



0 
7 



Floor 



2 ad floor 
3rd floor 



4 
3 



Bilingualism per floor 



Bilingual 
(English and Cantonese) 



Monolingual 
(English) 



2 »<S floor 

3 rd floor 



0 



4 
2 
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Supervisory and Licensed Staff Profiles 

Initially ten supervisory and licensed staff were chosen. Eight participated 
eventually. 



Gender 

Male 1 
Female 7 

Status 

DON 

Case-Mix Nurse 
Supervisor 
LPN/RN 
SDC 

Mother Tongue 

English 5 
Other 3 

Bilingualisxn (English and Mandarin) 

Bilingual 3 
Not bilingual 5 



1 
1 
2 

3 
1 



277 

9 

ERIC 



Learning of P articipants 



Do participants master the skills that are critical to the delivery 
of patient care of high quality ? 

Methodology 

1. Identifying key competencies or skills: 

Participants in the program, supervisors, nurses, and the staff 
development coordinator were asked to identify those functional uses of 
English that contribute directly to the quality of patient care. Their 
suggestions were used to compile an inventory. After that, participants, 
nurses, and supervisors indicated which uses of English they perceived as 
critical. Next, the inventory was compared with the language 
requirements specified in the skills section of the National Nursing 
Assistant Examination. The correspondance was almost one-to-one. 

2. Designing Simulations and Performance Standards: 

For each of the critical skills, a classroom-based simulation was designed. 
If possible, the actual tasks were performed. Three standards of 
performance were developed to describe the participants' peformance. 

1 * the participant does not know how to use English for this 

purpose 

2 - the participant has developed some skills but the use of English 

is minimal. The oral communication is not always 
comprehensible or successful. Single word sentences are 
overused. Uses of readin g, writi ng * nd numeracy are still 
incomplete or inaccurate. The style and register of language is 
still be inappropriate to the function. Bc^yjaaguagfi is not 
always comprehensible. 

3 « the participant has developed the skills necessary to perform 

the function. Communication is successful and comprehensible 
but not necessarily grammaticaUy correct. 
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Findin gs 



Individual Achievement 



JBof competencies attained 
3 2 1 

Participant 



1 


66.6 


13.4 




2 


733 


26.7 




3 


66.6 


13.4 




4 


60.0 


20.0 




5 


50.0 


30.0 


20.0 


6 


56.6 


43.4 




7 


66.6 


30.1 


3.3 


6 


60.0 


36.7 


3.3 


9 


70.0 


26.7 


33 


10 


63-3 


13.4 


3-3 


11 


70.0 


16.7 


13-3 


12 


73-3 


13-4 


13.3 



Average $ of competencies attained: 7 1.35 
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Group Achievement 



& of participants who 
attained competencies 

Competencies 3 



1 




100 


2 




633 


3 




91.6 


4 




633 


5 




91.6 


6 




56.3 


7 


Si 


50 


6 




41.6 


9 




333 


10 




75 


11 




633 


12 




41.6 


13 




41.6 


14 




75 


15 




100 


16 




41.6 


17 




66.6 


ia 




100 


19 




75 


20 




100 


21 




25 


22 




56.3 


23 




333 


24 




50 


25 




91 : 6 


26 




100 


27 


r 


63-3 


26 




91.6 


29 




63-3 


30 




91.6 
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Has the number of positive resident/nursing assistant 
interactions increased ? 

Methodology 

1. The perspective of residents was solicited by including these questions 
in the resident interview guide: 

- Does the nursing assistant speak with you during bathing, dressing, 

eating, toileting, or walking ? 

- Does he/she explain what he/she will be doing with you ? 

- Has talking with him/her 

0 gotten better <& gotten worse 0 remained the same 

3 

2. Nursing assistants self -assessed their interactions by rating their skills 
t to communicate with residents. Positive interactions were inferred from 
' the language requirements specified in the skills section of the National 

Nursing Assistant Examination. Interactions included: 

1- knock/ask for permission to enter 

2- greet the resident using his/her preferred form of address 

3- say name and job 

4- offer help and services 

5- say what will happen/state reason for coming 

6- explain procedures 

7 - instruct the resident speaking clearly and slowly 
6- ask about toileting needs 

9- encourage the resident to eat, exercise, and participate in 
activities 
10- indicate lack of understanding 
1 1 - apologize if necessary 

12- ask for help 

13- end a conversation inabrubtly 

14- give warnings in case of danger 

15- respond to requests, complaints, and concerns 

16- ask for feedback (eg. about comfort, clothing preferences, 

water and food temperature, order of feeding, etc.) 



o 
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Participants used this rating scale: 



1 « I don't know how to 

2 « I can but I dont 

3 * I do somewhat 

4 - I do fine 

Findings 

Resident perspective 



Participant Do interactions take place ? Are procedures explained ? 



1 


yes 


yes 


2* 


yes 


yes 


3* 


yes 


yes 


4 


yes 


yes 


5 


yes 


sometimes 


6 


? 


? 


7 


yes 


yes 


a 


yes 


yes 


9 


yes 


yes 


10 


yes 


yes 


11 


yes 


yes 


12 


yes 


yes 



Yes = 91.6 & Yes =63.3* 

Sometimes * 6.3 £ 
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Have positive interactions increased /improved, decreased /gotten worse, 
or remained the same ? 



Participant Increased Decreased Remained the same Don't know 

1 yes 

2* yes 

3* yes 

4 yes 

5 yes 

6 yes 

7 yes 
6 yes 

9 yes % 

10 yes 11 

11 t : yes 

12 yes 



Increased 66.6 £ 

Decreased 6 3 £ 

Unchanged 6.3 % 

Don't know 16.6 % 
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Participant perspective 
Individually 

Participant $ Positive interactions NA involves in 

4 3 2 1 NA 

! 93.75 6.25 

2 68.75 31.25 

3 75.00 12.50 12 50 

4 100.00 

5 43.75 56.25 

6 68.75 18.75 12.50 

7 68.75 12.50 12.50 6.25 

8 100.00 

q 18.75 50.00 31.25 

10 31.25 62.50 6.25 

11 (/& 56.25 37.50 

12 18.75 68.75 12.50 



Average 57.81 
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Has the number of positive interactions increased ? 



Group 

Interaction 

4 



1 


50.00 


2 


91.66 


3 


66.66 


4 


50.00 


5 


66.66 


6 


25-00 


7 


41.66 


6 


6333 


9 


66.66 


10 


66.66 


11 


66.66 


12 


50.00 


13 


41.66 


14 


66.66 


15 


25-00 


16 


66.66 



i 



% of participants/skill level 



3 


2 


1 


16.66 


3333 




6.33 






3333 






50.00 






% 33-33 






U 56.33 


6.33 


6.33 


50.00 


6.33 




16.66 






3333 






25.OO 


6.33 




3333 






33.33 


6.33 




33-33 


16.66 


a.33 


25.OO 


6.33 




33.33 


16.66 


16.66 


16.66 


6.33 


633 
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Has the number of misunderstandings decreased ? 

Resident Perspective 

- Does the resident understand the nursing assistant ? 

- Does the nursing assistant understand the resident ? 

Participant Understands NA ? NA understands resident ? 



2 
3 



well 
well 
well 



well 
a little 
weU 



4 

5 
6 
7 
6 
9 
10 
11 
12 



well 
well 
weU 
weU 
weU 



not always 



sometimes 



well 
weU 
well 



well 
well 
weU 
well 
weU 
NA 
weU 
well 



Well 



91.6JS 



75S 
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Participant perspective 



- Do misunderstandings take place ? 

. Has the number decreased, increased, or remained about the 
same ? 

Participant Talce place ? Decreased ? Increased ? Remained the same ? 



1 


no 


yes 


2 


no 


yes 


3 


no 


yes 


4 


no 


yes 


5 


no 


yes 


6 


no 


yes 


7 


no 


na[ 


a 


no 




9 


sometimes 


yes 


10 


yes 


NA 


n 


yes 




12 


yes 


yes 



9 
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yes 



yes 



66.6$ (no) 66.655 1&.68 
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Job Performance or Transfer o f Learning to the !ob 



Do nursing assistants properly use English to report changes in 
the condition of residents ? 

Findings 



Resident Perspective 
Participant 



Does he/she relay messages /report changes ? 



1 

2* 

3* 

4 

5 
6 

7 

6 

9 
10 
11 
12 



yes 
yes 

? 

% yes 

\ I don't know 

yes 

? 

yes 
yes 
yes 
yes 

I always report directly 



Yes = 66.6 % 



i 



9 
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Perspective of licensed and supervisory staff 



Participant Does he/she report to you ? 





DON 


Nurses 


SDC 


Supervisor 


1 


yes 


yes 


yes 


yes 


2 


yes 


yes 


yes* 


yes 


3 


yes 


yes 


yes* 


yes*" 


4 


yes 


yes 


yes* 


yes* 


5 


yes 


yes 


yes* 


yes* 


6 


yes 


yes/fco some 


yes* 


yes* 


7 


yes 


yes 


NA 


yes* 


o 


yes 


NA 


yes* 


yes* 


9 


NA 


NA 


NA 


yes* 


10 


NA 


yes 


NA 


yes* 


11 


NA 


NA 


No 


yes* 


12 


No 


No 


yes* 


yes* 



Directly in English to DON 66.6 % 

Directly to English only nurses 66.6 % 

* « reports in Cantonese /Mandarin 

NA ■ data are not available in most cases because informants 
did not know who the participant was. 
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job Performance or Transfer of Learning to tHe Tot> 

Do nursing assistants properly read and fill out work-related 
documents in English ? 

Methodology 

An exercise was designed to assess the participants' abilities to read and 
fill out work-related documents. The exercise was based on an English only 
flow sheet. Participants were asked to enter information from a written or 
taped narrative in their first language onto the flow sheet. Participants 
needed to enter 50 items of information. A scoring system was developed. 

Findings 



There was no concensus amongst participants, licensed, and supervisory 
staff on which codes should be used to enter certain kinds of information. 



Score in %\ 



Participant 



2 
3 

4 

5 

6 

7 
6 
9 
10 
11 
12 



74.0 

65-3* 

65.3* 

76.6 

72.6 

60.6 

653 
62.6 

64 

61-3 
70.6 
62.6 
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Organizational Performance 



Has SCM changed the way it operates in order to most 
effectively make use of the nursing assistants' skills ? 

Methodology 

Only one question was included to solicit answers to this outcome question. 
Participants were asked to give their perspective on how supervisory and 
licensed staff responded to their increasing language ability. 

Finding s 

Participant Do English speaking supervisory and licensed staff pay 

mo^e attention to NA's reports ? 



1 


NA 


2 


yes 


3 


yes 


4 


yes 


5 


yes 


6 


yes 


7 


NA 


6 


yes 


9 


NA 


10 


NA 


11 


somewhat 


12 


I don't know 



Yes = 50 % 

Comments 

The outcome study data -gathering instruments did not solicit sufficient 
information in this area. SCM has made use of the increasing skills of 
nursing assistants in ways that the data -gathering instruments did not 
address (e.g. sending participants to hospital to accompany resident, 
scheduling across floors, giving nursing assistants administrative tasks 
etc.) 
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Organizational Performance 



Has the organization changed ? 
Methodology 

To find out if the program has impacted on the performance of SCM as an 
organization, these questions were included that may enable the 
evaluation team to clarify the link between the participants* ability to 
report changes in condition and SCM's ability to achieve its goals: 

- Do N As report directly ? 

- Have their reports enabled licensed and supervisory staff to 

adjust the patient care plan ? 

- Have these adjustments/, improved the condition of residents ? 

Ti 

Perspective of licensed and supervisory staff 
Participant Did the report enable you to adjust the patient care plan ? 





DOW 


Nurses 


SDC 


Supervisors 


1 


yes 


yes 


yes 


yes 


2 


yes 


yes 


yes 


yes 


3 


yes 


yes 


yes 


yes 


4 


yes 


yes 


yes 


yes 


5 


yes 


no 


yes 


yes 


6 


yes 


sometimes 


yes 


yes 


7 


yes 


yes 


yes 


yes 


6 


yes 


NA 


yes 


yes 


9 


NA 


NA 


NA 


yes 


10 


WA 


yes 


NA 


yes 


11 


NA 


sometimes 


no 


yes 


12 


no 


NA 


sometimes 


sometimes 


Yes 


66.6 % 


50 % 


66.6 % 


91.6 % 



Have these adjustments led to improvement of the resident's condition ? 
( Have the reports helped the nursing home achieve its goals of restorative 
care ?) 

Participant Improvement ? 





DON 


Nurses 


SDC 


Supervisors 


1 


yes 


yes 


yes 


yes 


2 


yes 


yes 


yes 


yes 


3 


yes 


yes 


yes 


yes 


4 


no 


yes 


yes 


yes 


5 


no 


no 


yes 


yes 


6 


no 


$ yes 


no 


yes 


7 


no 


u yes 


NA 


yes 


a 


' no 


NA 


yes 


yes 


9 


NA 


NA 


NA 


yes 


10 


NA 


yes 


NA 


yes 


n 


NA 


yes 


no 


yes 


12 


no 


NA 


sometimes yes 



Yes 25JS 666JS 50% 100% 
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Participant Perspective 

Participant Shares language besides English ? Reports directly m English ? 



1 


no 


yes 


2 


no 


sometimes 


3 


yes 


no 


4 


yes 


sometimes 


5 


yes 


00 


6 


yes 


sometimes 


7 


yes 


no 


8 


yes 


no 


9 


no 


no 


10 


yes 


no 


11 


yes 


no 


12 


yes 


no 



?5*&<yes)t 8.3% (yes) 



Participant Less dependent on interpreter ? 



1 


yes 


2 


yes 


3 


yes 


4 


yes 


5 


somewhat 


6 


yes 


7 


yes 


8 


yes 


9 


yes 


10 


yes 


11 


nothing 


12 


somewhat 



Have reports led to adjustments of plan ? 
yes 

I don't Know 
I don't fcnov 
1 don't know 
no 

I don't hnov 
no 
no 
no 

yes 

I don't know 
I don't hnov 



75^ (less) 



16.6% (yes) 
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Participant 



Hoticed improvements in resident's condition as a result of reports ? 



1 


no 


2 


I don't knov 


3 


I don't knov 


4 


HA 


5 


HA 


6 


I don't knov 


7 


no 


8 


HA 


9 


HA 


10 


yes 


11 


2 don't knov 


12 


1 don't knov 



8.3 %(yes) 
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Or ganizational Performance 



Has SCM changed the way it operates in order to most 
effectively make use of the nursing assistants' skills ? 

Methodology 

Only one question was included to solicit answers to this outcome question. 
Participants were asked to give their perspective on how supervisory and 
licensed staff responded to their increasing language ability. 

Finding s 

Participant Do English speaking supervisory and licensed staff pay 

more attention to NA's reports ? 



1 


:i WA 


2 


yes 


3 


yes 


4 


yes 


5 


yes 


6 


yes 


7 


NA 


a 


yes 


9 


NA 


10 


NA 


11 


somewhat 


12 


I don't know 



Yes = 50 % 

Comments 

The outcome study data-gathering instruments did not solicit sufficient 
information in this area. SCM has made use of the increasing skills of 
nursing assistants in ways that the data-gathering instruments did not 
address (eg. sending participants to hospital to accompany resident, 
scheduling across floors, giving nursing assistants administrative tasks 
etc.) 
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Additional Data: Changes in the Use of Interpreters and 
Translations 



Participant Perspective 

Participant Do English speaking staff need an interpreter to communicait villi NA's ? 



1 


no 


2 


no 


3 


no 


4 


no 


5 


yes 


6 


no 


7 


no 


8 


no 


9 


yes 


10 


no 


11 


yes 


12 





66.6% (no) 



Perspective of supervisory and licensed staff 
Participant Become less dependent on interpreting ? Need interpeter ? 





DON 


Nurses 


DON 


Nurses 


1 


yes 


yes 


not usually 


no 


2 


no 


no 


no 


not usually 


3 


yes 


yes 


no 


no 


A 


yes 


yes 


no 


not usually 


5 


no 


yes 


yes 


not usually 


6 


no 


sometimes 


yes 


sometimes 


7 


yes 


yes 


no 


not usually 


8 


yes 


NA 


not usually NA 


9 


NA 


NA 


NA 


NA 


10 


NA 


yes 


NA 


not usually 


11 


NA 


yes 


NA 


not usually 


12 


no 


yes 


yes 


yes 




41. 69$ (yes) 


66.69&(yes) 


5095* 


66.695* 



* = no + not usually 
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Participant 



Interpret less for NA ? 
Supervisor 



Depend kss on interpreting ? 
Supervisor 



1 NA NA 

2 yes yes 

3 yes yes 

4 yes yes 

5 yes NA 

6 yes yes 

7 yes yes 

8 yes NA 

9 yes yes 

10 yes yes 

11 yes NA 

12 yes NA 



91.6«<yes) 58.395 (yes) 

8.3% (xio&applicable) 41 .795 (not applicable) 
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Personal Development of Nut sintf Assistants 

Have nursing assistants benefitted from tne ESOL program in 
ways not directly related to their jobs ? 

Methodology 

1. Based on a review of previous non work-related outcomes, several 
possible areas were identified. Participants were asked to respond to these 
categories in the self -assessment questionnaire by indicating those areas 
where the program helped them. Open-ended questions were included to 
solicit additional information. Supervisory and licensed staff were asked if 
they had observed any non work-related gains as well. 

2. To measure any possible^gains in general proficiency (i.e. non work- 
related) the BEST-test was administered. Scores provide information about 
the participants' abilities to use English for survival purposes in the 
community, a comparison of pre- and post testscores indicates the overall 
proficiency gain of participants. Prescores refer to the first time the BEST- 
test was administered in the program. It should be noted that certain 
participants had already received a substantial amount of instruction at 
that time. 

Findings 

Participant Perspective 



* of participants 



Kind of benefit 



Better understand American culture 
5 etter prepared for alternative 
employment 

Better prepared for future education 



Made nev friends 



2 
A 



5 



Other 



6 



Zdd 



Perspective of Supervisory and Licensed Staff 



* of times reported 

Kind of Benefit 

Became much more confident 1 3 

Became much more social 2 

Became more independent 1 
Speaks more/Takes more initiative/ 

Uses English voluntarily 12 

Increased her self-esteem 1 

Learned some Chinese and practices 1 
with residents 

Saves time/Helps me save time 1 

Mates steady progress 2 

Ti 

Proficiency 

Participant Pre Hrs. of Instr. Post Hrs. of Instr. Gain 



1 


NA 


NA 


NA 


316 


NA 


2* 


43 


164 


64 


440 


21 


3* 


67 


134 


72 


392 


5 


4 


43 


166 


55 


372 


12 


5 


26 


176 


40 


456 


12 


6 


19 


102 


40 


322 


21 


7 


21 


94 


36 


266 


17 


6 


0 


0 


53 


236 


53 


9 


1 


0 


NA 


255 


NA 


10 


NA 


0 


NA 


216 


NA 


11 


NA 


0 


NA 


64 


NA 


12 


12 


0 


62 


146 


50 



Average gain/Hrs. of Instr. Received between pre /post 

23.67 (*26.06*)/ 224.5 hrs. 
Maximum Score = 65 
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Su ggestions 



What can the ESOL program and SCM do to help NAs improve 
their skills ? 



- keep speaking English 

- encourage more English 

- put the stress on using English 

- encourage NAs to express their lack of understanding instead of nodding 

- use flow sheets and other documents more often to increase the NA's 

understanding of providing quality information 

- prepare NAs for higher ed 

- increase hours of instruction 

- make program more accesibl^to all staff 

- offer a Mandarin class for English speaking staff 

- correct the NAs' English 

- fine the way it is 

- provide bilingual documents, write everything side by side English and 

Chinese 

- concentrate on job preformance and specifics 

- reduce amount of Chinese translation 

- use more real' roleplays in class 

- try to use English to communicate before using another language 
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Which data have not been processed yet ? 

1. ratings of language skills by participants and supervisory and 

licensed staff 

2. ratings that show how the program has impacted on the skills of 

workers 

3. frequency of reporting to English speaking supervisory and 

licensed staff 

Which data can be further analyzed ? 

Basically all according to groupings the team desires. 



1 
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SIMULATIONS 



' SIMULA! ION v 1 



WHAT WE ARE LOOKING AT 



Did you ? 

1. knock and/or ask for permission to enter ? 

2. greet the resident using the resident's name and/or preferred form of address ? 

3. identify yourself by name and job (e.g. I am Johan. I am your nurse aide today). 

4. offer help or your services (e.g. Can I help you ? Can I get you anything ?) 

5. say why you came (e.g. I'm going to make your bed. Is that n K ?) 

6. instruct the resident speaking clearly and slowly ? (e.g. Could you turn your 

head towards me, please ?) 

7. ask about toileting needs ? 

8. encourage the resident to participate in exercises or activities ? 

9. let the patient know when you didn't understand him/her ? 

10. apologize if necessary ? (e.g. I'm sorry. I didn't mean to hurt you.) 

11. give a warning in case of potential danger ? (e.g. Watch out, Mr. X) 

12. respond to the resident's questions, concerns, or complaints ? 

13. look the patient in the eyes as much as possible ? 

14. ask for feedback (e.g. Is this fine ?) 

15. relay the messages the patient gave you to the CN ? 

16. report to the nurse aide what you did/observed ? 

17. begin, maintain, and end a conversation while shaving/ambulating ? 
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OUTGONE STUDY 



Are participants able to use English for these purposes that are essential to the 
delivery of patient care ? 



SIMULATIONS 



Simulation 0 1 

a 

Instructions for NA: Please, shave this resident using an electric razor. 
Start outside the room and think of this room as the resident's room. 
The patient's name is Joe Smith. After you finish, please report to 
the charge nurse. 

Instructions for patient: Do not initiate a conversation. Cause a situation 
that will make the NA apologize and give a warning (e.g. Ouch, bend over) 
Respond to the NA as you please. At the end, ask the NA to relay these 
two messages; 1) a skin rash on your arm, 2) you want to play Bingo 
todav so you would like to know if you can have your medicine at 2 instead 
of 3. 

Instructions for CN: listen to the NA's report. Ask for clarification if necessary. 
Do not initiate the conversation/report. 



Simulation # 1 

bis 

Instructions for NA: Please, ambulate this patient from his/her w/c to the 
door and back. 

The patient's name is Mary Donovan or Bill Jones. Report to the CN 
afterwards. 

Instructions for patient: Do not initiate a conversation. Pretend you will 
step in a puddle or trip over something to solicit a warning. Create 
a situation where the NA will have to apologize (e.g. Ouch, you're 
hurting me). Mention a few times you are short of breath today and 
breathe accordingly. 

Instructions for CN: see above in Simulation 0 1 a. 



Materials Simulations 1 a + 1 bis: wheelchair, electric razor, brrom , or 
puddle. 
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Coded Use/Objective 



Description 



1 
2 
3 
4 
5 
6 
7 
8 
9 

10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
?3 

24 
25 
2C 
27 

28 

29 

30 



ERIC 



knock/ask ;or permission to enter 
greet usinn name/preferred form of addr 
identify self by name /job 
offer help/services 

state reason for coming/what will happe 

instruct resident speakino clearly/slow 

ask about toiletina needs 

encouraqe resident 

indicate lack of understandino 

apologize if necessary 

ask for help 

end conversation inabruptly 

qive warnings in case of danger 

respond to resident requests /complaints 

maintain eyecontact as much as possible 

initiate and maintain a conversation 

ask for feedback 

follow oral instructions 

ask for clarification of instructions 

follow instructions over P. A. -system 

report chancres in condition orally 

relay resident messages 

report accidents, unsafe conditions, 

mistreatment, neolect, and abuse oral! 
report breakdowns and shortaoes 
call in sick/late 
infer work time from schedule 
infer instructions for servina fror 

diet cards 
follow written signs, warninos 

infer tasks from dailu posted 

nursina sheet 
measure and record vital sinns 

and intake/output 
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COTCCHE STUCK JOB SIMULATIONS 



IN>IVIDUftL AH) SO? ACHIEVEMQ7T 





1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


]] 


!2 


11 

1 mJ 






15 


17 


18 


;19 


; 20 


21 


22 


23 


24 


25 


26 


27 


26 


29 


30 


a 
1 

of 4>j^c4«\#r«j 


PARTICIPANTS 
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At 7am, I told Kate "it's tine to get up, M She didn't wu:.t 
to get up. She said: "do other people first." But when I said 
"please, " she said OK. I helped her to get dressed, and j.ut the 
socks and shoes on for her. I helped her walk to the bat he cos. 
She had a shower. I scrubbed her back for her because she 
couldn't reach there by herself. I shaved her <»nJ cut he: fir.ger 
r.ails. She needs a shave and has her finger nails cut wttrkiy. 
She combed her hair by herself. 

I helped her to put on her clothes, and did her buttons on 
the back. At 10am, May Chen and I took her for a 20-£eet walk. 
She never likes to walk. When we encouraged her, she said "just 
a little" and she did. Most of the day sh* sat in a wheelchair. 
When she needed to get out of her bed or wheelchair, someone had 
to help her. 

I took her to the bathroom every hour to empty her b2a.ldt;r, 
because she is incontinent. Although she k««pt saying "I vent M or 
"I have none," I knew that :,he didn't. Right after her breakfast 
and lunch, I took her to the bathroom to have her stool. We all 
have to keep this in mind, if someone forgets to do this right 
after a meal, she will have to do a lot more work. 

She complained about her pains here and there or said "I'm 
dying" again several times. But she was basically a sober-minded 
person, and she's friendly and very nice. 
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FLOW SHEET EXERCISE 



SCORING SHEET 



1. Month 

2. Year 

3. Resident name 

4. Resident room # 

5. Initials 1 

6. Initials 2 

7. Signature 

8. Shift l=D) 

9. Day (=Tuesdav) 

10. Bed Rath 

11. Partial 

12. Whirlpool/Shower 
13* Shdv@ 

14. Mouth/Denture Care 

15. Mouth Special Care 

16. Nails 

17. Grooming 

18. Dressed Day/Night 

19. Socks and shoes only 

20. Not dressed 

21. Ambulate 

22. Distance 

23. Device 

24. Geri/Wheel Chair 

25. Transfer 

26. Device . 

27. Bad or W/C Position 

28. ROM 

29. Bladder Continent Freq. 

30. Oncontinent Freq. 

31. Toileted 

32. Bowel Continent Freq. 

33. Incontinent freq. 

34. Toileted freq. 

35. Bladder Retraining 

36. Bowel Retraining 

37. Catheter Care 

38. Wandering 

39. Hoarding 

40. Noisy 

41. Verbal abuse 

42. Physical abuse 

43. Uncooperative 

44. Other 

45. PiVv<?ntive Skin Care 

46. Elbow/Heel Protector 

47. Brace/Splint 

48. Sheep Skin 

49. Other (abdominal pain) 

50. Other column * 



2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 

2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 

2 
2 
2 
2 
2 
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3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 

3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
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TOTAL <=T) 
SCORE: T X 



QUESTIONNAIRE SUPERVISORY AND LICENSED STAFF 
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Outcome Study 
SCM Workplace Education Program 

Questionnaire to be completed by English speaking 
supervisory end licensed staff 

Your position: Dale: Name o/nuraing assistant: 

PART 1 

Reporting changes in residents' condition 

1. Do you share a language with him/her other than English ? 

O Yes 0 No 
If yes, which one(s) ? 

2. How long have you worked with him/her ? 



3. Does he/she report to you 

0 directly in English ? 
O directly in Chinese ? 
O through an interpreter ? 
0 not at all 

4. How often does he/she report to you ?(e.g. once a week during 
patient care rounds) . 

5. Have his/her reports enabled you to adjust the patient care 
plan as necessary ? 

0 Yes 00 No 0 

If yes, have these adjustments had an impact on the resident's 
condition ? 

0 Yes 0 No 0 

Please, explain your answer: 



6. Do you interpret for him/her ? 

0 Yes 0 ::o 
If yes, hcv; often ? icq. once a weeV:)___ 

7. V.'oulc: you say he/she has cccor.e less ^eocndent on you to 
interpret for him/her ? 

0 Yes 0 ?.*o ° 
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8. Do you need an interpreter or written translations to instruct 
him/her ? 

0 Yes 0 !Jo 0 Hot usually 
If yes, under what circumstances 



9. Have you become less dependent on interpreters or translations 
to communicate with him/her ? 

O Yes 0 No 

10. Please, record any other comments related to his/her ability 
to report changes in the residents' conditions. 

PART 2 General Comments 

1. Do you generally know who is in the ESL procrram ? 

0 Yes 0 No 

2. How would you rate his/her ability to use Enolish on the job ? 

0 Excellent 0 Good 0 Average 0 Somewhat weak 0 Weak 

3. Have you noticed any additional changes since he/she enrolled 
in the ESL program ? (e.g. more confident) 



4. According to you, how much has the ESL proaram contributed to 
improving his/her languaae skills ? 

0 :io idea 0 A lot 0 A little 0 I'.ot at all 

5. How can the ESL program better help nursing assistants improve 
their language skills ? 

6. Wiat else miqht be done by SCM, by supervisors, and by nursino 
assistants to help nursing assistants use English to perform 
their jobs ? _ ... 

THANK YC J FOR FILLING OUT THIS OUESTICNNAIRE. IF YOU HAVE OUESTIOMS, 
PLEAEE CONTACT BARBARA IRVING, HSIAO CHANG, OR J OK AN UVIN. 



314 



¥ 



RESIDENT INTERVIEW 
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Ovl€tin# Study 
SCM Workplace Education Program 
Interview Guidelines 
Resident Interview 



Name of participant: 



Name of resident : 



Your name: 



Floor 
Dale: 



INSTRUCTIONS: Explain the purooseofthetatenfewtotheresidentfe.g. "SCMwantstofindouthow 
vefnureinflaaslatentacanuae&i^shtotaflcv^hyou.n Ask if it Is fine thet you would take notes. 
Mention thai the results of the WefMewvdl be kept confidential. Use one ormore of these questions to 
irvTiateofrneirUeJn a brief conversation. Record the resident's e/iswerbychecWngtheansverthat comes 
closest to what the resident says. Feel free to record responses more eiab calefy 



1. Does he/she speak English with you during bathing, dressincr, 
eating, toileting, or walking ? 

O Yes O No 0 



2. Does he/she explain what he/she will be doing with you ? 
O Yes O No o 



3. Does he/she give your messaces to the nurse ? 
0 Yes 0 No 0 



4. How well do you understand him/her when she/he is talkino 
with you ? 

0 not at all 0 a little 0 well 



5- How well does he/she understand you when you are talking to 
him/her ? 



If 4 + 5 are hard, why do you think that is ? 

0 doesn't know English very well 

0 doesn't understand her job 

0 

6. Overall, has talking with him/her 



0 gotten better ? 0 aotten worse ? O stayed about the same ? 



0 



0 not at all 



0 a little 



0 well 



COMMENTS 
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QUESTIONNAIRE NURSING ASSISTANTS 
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OUTCOME STUDY 
QUESTIONNAIRE FOP. NURSING ASSISTANTS 



Your name: ____ . Date: 

PART 1: INTERACTIONS WITH RESIDENTS IN ENGLISH 

INSTRUCTIONS: Please check the answer which most closely describes 
your ability to perform these job tasks. Check only ONE answer for 
each item. 

KEY : 1 = don't know how to 2 - can but don't 3 = do somewhat 
4 ■ do fine 



1. knock/ask permission to enter 

2. greet resident using name and/or 
preferred form of address 

3. eay my name and job 

4 . offer help and services 

5. say what will happen/why I came 

6. explain procedures 

7. instruct resident speaking clearly 

and slowly (e.g. vital signs, dressing, 
feeding, etc.) 

8. ask about toileting needs 

9. encourage resident to eat, exercise, 
and participate in activities 

10. let resident know when I don't 
understand 

11. apologize if necessary 

12. ask for help 

13. end a conversation inabruptly 

14. give warnings in case of danger 

15. respond to resident questions, 
complaints, and concerns 

16. ask for feedback (e.g. comfort, 
clothing preferences, 

water temperature) 
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INSTRUCTIONS: Please answer these questions. 

17. Overall, how would you rate your ability to perform the above tasks 
in English ? 

0 Excellent O Good C Average 0 Needs some improver.ent 
0 Needs a lot of improvement 0 „ 

18. What could be done to help you improve your job-related English 
skills ? 



PART 2 MISUNDERSTANDINGS WITH ENGLISH SPEAKING RESIDENTS 

!. do languaoe misunderstandings regularly take place between you 
and the residents ? 



C No 0 Yes 

If yes, why, with whom, and about what ? 



2. Has the number of misunderstandings 

O Increased ? C Decreased ? 0 Remained about the same ? 

3. How do you normally avoid or resolve misunderstandings ? _ 



. What has helped you to avoid or resolve language misunderstandinos ? 
(For example: inservices, ESL workshops, advice from a co-worker .) 
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PAttT 3 REPORTING CHANGES IN THE CONDITION OF RESIDENTS 

1. Do you share a language with your supervisor other than Enqlish ? 

O Yes XJ No 

If yes, which one (s) ? 

2. Does your supervisor know that you are in the ESL program ? 

0 Yes O No 0 1 don't know 

3. How long have you worked with your supervisor ? . 

4. Do you report to your supervisor 

<D directly in English ? 
O directly in Chinese ? 
o through an interpreter ? 

5. How often do you report to your supervisor ? { 

6. Do you report to English speaking charge nurses and other nursintr 
staff (e.g. Director of Nursing) ? 

0 Yes O No 

7. If yes, do you report 

0 directly in English ? 
u through an interpreter ? 

8. Does your supervisor interpret for you ? 

O Yes 0 No 
If yes, how often ? _ 

9. Who else interprets for you ? ^ 

10 . Have you become less dependent on your supervisor to interpret 
for you ? 

11. Have your reports to English speaking staff enabled your supervisor 
to adjust the patient care plan as necessary ? 

0 Yes 0 No 0 1 don't know 

If yes, have you noticed changes in the resident's condition 

as a result of these adjustments ? 

0 Yes 0 No 0 1 don't know 

12. Please give an example: | ... M 
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13. Does your supervisor need an interpreter or written translations 
to tell you what to do or ask you a question ? 



0 Yes 0 No 
If yes when ? 



14. Do English speaking charge nurses and nurses need an interpreter 
or written translations to tell you what to do or ask you a 
question ? 

0 Yes P No 

If yes, when ? - •- 



15. Do you feel that English speaking nursing staff pay more attention 
to what you have to say about care of residents ? 

0 Yes 0 No 
Please, explain your answer: , . — 

16. What should be done to improve your ability to report changes in 
the resident's condition ? ■ 



t»X: 4 OTHER OUTCOMES 

INSTRUCTIONS: The following questions ask how the ESL program has 
helped you with your personal goals. Please check ONE answer for 
each question. 

HAS THE ESL PROGRAM HELPED YOU No Somewhat A lot 

1. make new friends ? 

2. understand American culture ? 

3. get a better job ? 
A. get more education ? 

5. become a U.S. citizen ? 

6. improve your health ? 

6. Please describe any other ways the ESL praram h?»s helped you: 



9. What can the ESL program do to better help you ? 
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CASE STUDY: 
OUTCOME STUDY AT UNITED ELECTRIC CONTROLS 



Background: United Electric Controls Company, Watertown 

Since 1987 this successful family-owned manufacturer of industrial sensors and 
temperature control devices has overhauled its management and production 
technologies according to the principles of "continuous improvement." 
Motivated by severe profit losses, it abandoned traditional "large lot" 
manufacturing methods in favor of the techniques that stress employee 
participation in the production process. In order to facilitate these company- 
wide changes, management offered ESL instruction to its limited- English- 
proficient workers, approximately 45 of its 350 employees, in 1990 United 
Electric Controls Company (UE) won the prestigious Shingo prize for 
excellence in manufacturing. 

In the early 1980s, unable to find native-English-speaking people for assembly 
work, UE hired a small group of Cambodian refugees and conducted its first 
ESL program with them. The success of this year-long program resonated a 
few years later with the need articulated by one of the new employee driven 
"action groups" to provide English language instruction for employees. 
Dramatic changes in the production process required alt workers to employ 
more written and oral communication skills, to speak up in their action groups 
and take initiative with solving problems. Umited-English-proficient workers 
had double trouble learning the new ways. They not only had to learn the new 
methods themselves, they had to learn the English language vocabulary that 
would allow them to learn the new methods. 

The decision to overhaul production methods brought the need for English- 
language instruction to the fore. But all employees at UE were asked to learn 
something new. They were being asked to team how to view their work as an 
opportunity for continuous improvement of the production process, which 
demanded an atmosphere of study and practice. The action group which 
identified the need for ESL instruction was acting in accordance with the new 
guidelines not to be satisfied with the status quo. 

ESL classes meet for a total of six hours per week. The company provides full 
release time for 45 students. 

The barriers to success of the UE ESL program came primarily from native- 
English-speaking employees who at first resented.the extra attention the 
limited-English-proficient workers received. Slowly, over time, with the help of 
an excellent teaching staff and the support of upper management, these 
barriers have fallen away. Bruce Hamilton, Vice President of Manufacturing, 
wrote: "Lessons learned at United Electric deal not so much with the results 
often associated with "just in time" manufacturing or even the techniques by 
which these results are achieved. Rather, they revolve around a struggle to 
understand which of the manufacturing traditions that have served the company 
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so well for sixty years must be retained, and which must be changed." The ESL 
program continues to be an integral part of the overall strategy to change and 
continually refine the production process. Without it, old and new employees 
who have limited English-language skills would not be able to participate fully 
in these changes. 

Description of the evaluation process: 

The make-up of the evaluation \mam: The UE team was composed of a vice 
president for human resources, a shop-floor supervisor, and two 
representatives of the education-providing institution, the Middlesex County 
Employment and Training Program. (One of these representatives served as 
coordinator of the program and the other functioned as instructor.) A student in 
the program also participated in early meetings of the team. 

The process: The UE team followed the ten-step process described in Section 
11 of the report of the MWEI Cycle 4 Evaluation. Ail members of the team 
showed an active interest in the project, although the major responsibility for 
gathering and analyzing data was left in the hands of the program coordinator 
and instructor. 

What was learned about the education program; See Section II, Step 8 of the 
report of the MWEI Cycle 4 Evaluation for a summary of the team's findings, and 
the attached data summaries. 

What was learned about the evaluation process; The active involvement of the 
UE management representatives was a major strength of the project. The 
supervisor participating on the team had been a prime mover in getting the ESL 
program started in the first place, and her interest was demonstrated in her 
active participation in team meetings. The vice president for human resources 
was also key to the process. He had helped the company map out a sweeping 
change toward a more-participatory form of management (of which the ESL 
program was one component), and he saw this outcome study as a way of 
assessing the impact which these changes were having on the company. By 
showing an interest in what the data said, he in effect provided an "audience" 
and purpose for the study. In terms of limitations, the education staff - like those 
in all other sites participating in the project - had limited time to give to the 
project, as they had other job duties to deal with. Another limitation was the fact 
that the student representative on the team was involved in only a limited way, 
in part because her English was limited but also because she had little 
background in the issues discussed by the team. This suggests that special 
attention be given to helping learners have meaningful roles on evaluation and 
planning teams. 
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Attached data-gathering Instruments, summaries of findings, and related 
documents: 

Data-gathering instruments: 

- Questionnaire for ESL Students 

- Questionnaire for Supervisors 

Summary of findings: 

The data are presented in a summary grid. This team did not produce a 
final report. 
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RESULTS FROM SUPERVISOR QUESTIONNAIRES: 


UNITED ELECTRIC CONTROLS. FALL 1991 
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Response codte 






















NE- Employee not expected to perform this tas 


k. 






NR= No response given to this Question. 








1= Employee can do this, but doesn't do it, 








2=Empiovee does this somewhat. 














3= Emlovee does this well 1 
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UNITED ELECTRIC CONTROLS 
E.S.L PROGRAM EVALUATION: 

QUESTIONNAIRE 
FOR 
SUPERVISORS 



5/6/91 



Name of supervisor completing questionnaire 

Signature of supervisor 

Date questionnaire completed 

Employee being evaluated 



INSTRUCTIONS: 

w U ' E ' wou,d Bke to know wnat im ^ icX ESL program has had on the above employee's 
job performance. This questionnaire wfll help U.E know how to improve the ESL prop/am. 

Please rate how well the employee can use English to perform the followhg tasks at this 
time. (If the employee doesn't normally have to perform a particular task, write "doesnl apply" in 
the space marked "Explain your answer.-) 

Results of this questionnaire wiU be kept confidential for you and the employee, so please 
be as accurate and honest as prasible. 
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Um of document!: 



1. The employe© understands the information contained in the yearly performance review. 

I dont know one way or the other. 

Employee canl do this at afl. 

Employee cj_do this, but doesn't do it. 

Employee does this somewhat. 

Employee does this well. 

I Explain your answer: 



2. The employee understands and uses "prints" required by the job. 

I donl know one way or the other. 

Employee cant do this at al. 

Employee pan do this, but doesn't do It. 

Employee does this somewhat. 

Employee does this well. 

Explain your answer: 



3. The employee understands the X-matrix and uses English to discuss information in the X-matrix 
aa it relates to die job. 

I donl know one way or the other. 

Employee canl do this at all. 

Employee £fin do this, but doesn't do it. 

Employee does this somewhat. 

Employee does thfe well. 

Explain your answer: 



CommunBesUng In English trttii other group*: 

4. In the plant, the employee uses English to communicate with groups whose mother tongue is 
different from the employee's. 

I donl know one way or the other. 

Employee canl do this at all. 

Employee an do this, but doesn't do H 

Employee does this somewhat. 

Employee does this well. 

Explain your answer: 
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Teamwork: 

5. The employee usee English to participate actively in CEDAC groups 

I dont know ont way or the otter. 

Employee cent do this at al. 

Employee can do this, but doesn't do it. 

Employee does this somewhat. 

Employee does this well. 

Explain your answer: 



6. The employee uses Engfish to participate actively in action centers. 

I dont know one way or the other. 

Employee cant do this at afl. 

Employee can do this, hut doesn't do it. 

Employee does this somewhat. 

Employee does this well. 

Explain your answer: 



7. The employee uses English to participate actively in production work cells. 

I dont know one way or the other. 

Employee cant do this at ail. 

Employee cajQ do this, but doesn't do it. 

Employee does this somewhat. 

Employee does this well. 

Explain your answer: , 



8. The employee uses English to participate actively in cross-training activities. 

I dont know one way or the other. 

Employee cant do this at ad. 

Employee can do this, but doesn't do it. 

Employee does this somewhat. 

Employee does this well. 



Explain your answer: 



9. The employee uses EngBsh to participate actively in the Valued Ideas Program. 



I dent (mow one way or the other. 

Employee cant do this at ai 

Employee can do this, but doesn't do it. 

Employee does this somewhat. 

Employee does this well. 

Explain your answer: 



Health end ■trfefy: 

10. The employee uses ergonomics to practice a "healthy work style" on the job. 

I don't know one way or the other. 

Employee cant do this at aft. 

Employee can do this, but doesn't do it. 

Employee does this somewhat. 

Employee does this well. 

Explain your answer: 



Additional eommsnts: 

11 . Overall, how would you rate the employee's ability to use English to perform the above tasks 
during the past six months? 

The employee is about the same as six months ago. 

The employee has improved a tittle bit. 

The employee has improved a tot 

12. Please sdd any other information (special accompfishments, problems, examples) about the 
employee's abilities to use English on the job which you think we should know about: 



13. If you have any suggestions for how the ESL program might be improved, please add them 
here: 



Thank you for your cooperation in completing this questionnaire. 



* 334 



QUESTIONNAIRE 

FOR 
E.S.L STUDENTS 
AT UNITED ELECTRIC CONTROLS 

5/6/91 



Name of employee 

Date questionnaire completed 
Signature of employee 



INSTRUCTIONS: 

U.E. would Bke to know how much the ESL procram has helped you on and off the job. 
You can help us by completing this questionnaire. 

Please answer the foBowing questions about how well you can now use English. At the 
end there is space where you can add other comments. 

Results of this questionnaire wifl be kept confidential. 



Ptoses rata) jfow ability/ to perioral the foU owing tesfcs at U.E.: 



Un of documents: 

1 . I understand the information contained in the yearly performance review. 

I cant do this at all. 

I can do ft. but I don't doit 

I do it. but not as well as I would like to. 

I do it we«. 

Explain your answer: 



2. I understand and use "prints'' required by my fob. 

I cant do this at ail. 

I can do this, but I don't do it. 

I do this, hut net as well as I would Bke to. 

I do this well. 



Explain your answer: 
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3. I understand the X-matrix and use English to discuss information in the X-matrix as M relates to 
my job. 



I cant do this at ail. 

I can do this, but I dont doit 

I do this, but not as wed as I would Ska. 

1 do this well. 

Explain your answer: 



Communicating In English with other groupa: 

4. In the plant I use English to communicate with groups whose mother tongue is different from 
mine. 

I cant do this at all. 

I can do this, but I don't do it. 

1 do this, but not as well as I would like. 

I do this well. 

Explain your answer: _ 



5. 1 use English to participate actively in CEDAC groups. 

I cant do this at aS. 

I can do this, but I dont do it. 

I do this, but not as well as I would like. 

I do mis well. 

Explain your answer: 



6. 1 use English to participate actively in action centers. 

I cant do this at all. 

I can do this, but I dont do h. 

I do mis. but not as well aa I would like. 

I do this well. 



Explain your answer: 



7. I use English to participate actively in production work cells. 



I cant do this at all. 

I can do this, but I dont do thia 



I oo this, but not as well as I would like. 
I do this wet). 
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Explain your answer: 



6. I use English to participate actively in cross-training activities. 

I cant do this at ail. 

I can do this, but I donl do this. 

I do this, but not as wetl as 1 would like. 

I do this well. 



Explain your answer: 



9. 1 use English to participate actively in the Valued Ideas Program. 



I cant do this at all. 

I can do this, but I don't do it. 

I do it, but not as well as I would Tike to. 

I do it well. 



Explain your answer: 



Health and Misty: 

10. 1 use ergonomics to practice a "healthy work style" on my job. 

I can't do this at all. 

I can do this, but I don't do it. 

I do H, but not as well as I would like to. 

IdoHweH. 

Explain your answer: 



Additions! comcnenCo about using Engtlsgi on the gob: 

11. Overall, how would you rate your ability to use English to hands the above tasks during the 
past six months? 

I am about the same as I was six months ago. 

I have improved a little bit. 
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I have improved a lot 



12. If you fed your fob-related English skflls have Improved, what haa been the causa of the 
Improvement? (Check afl answers that apply, ranking them in order of impact on you- skids, with 
"1" having the most impact) 

I've learned from using English on the job. 

The ESL dass has helped. 

I've learned from other experience (for example, another education program. 

tutoring from a family member, watching TV. etc.) Please be specific: 



Using English off the Job: 

P!sv*e> check ail of the foBowkig nanra which apply to yon: 

13. In adeftion to my ESL classes at U.E.. I have: 

Applied to adult education programs in the community. 

Been accepted into an adult education program in the community. 

Begun classes in an adult education program. 

Completed classes in an adult education prog-am. 

14. In addition to technical training I have received at U.E.. I have: 

Applied to a technical training program in the community. 

Been accepted into a technical training program. 

Begun dosses in a technical framing program, 

Completed classes m a technical framing program. 



15. Outside the job, I now am more comfortable using English: 

With my children. 

With neighbors and friends. 

With strangers on the street, at the market, etc. 

In my children's school. 

In understanding the news. 

In my place of worship. 

In community activities like a block club, etc. 



Final eotnmanfta: 

16. Please add any other information about your abilities to use English on or off the job which 
you think we should know about. 



Personal Information: 
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17. When old you start in the ESL program? 



16. Sines you first began, old you d-op out of the program for any amount of time? 

Ves (if "Yes." when dd you orop out and when did you return? 

) 



No 



19. How many months in total have you attended class? months 



20. Before the class began, what waa your skill level in; 

listening in EngBafr 

Good 

Medium 

Poor 



Speaking Enotish: 



Good 

Medium 

Poor 



Reading, in Sngjjsh,: 



Good 

Medium 

Poor 



Writing in Endish: 



Good 

Meolum 

Pooi 



Thank you for your cooperation in fifing out this questionnaire. 
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Employee 



Supervisor Comments (slightly edited) 
Question 91 

Employee 4 I. I help Nancy understand review. ' t - 

2. I feel can read, understand, as well. I take time to make sure. 

3 can do this with a little assistance. M 

4. ...helped her read and understand everything - j. 

5. .. .fully understands. . . 



Q. 2 



Q. 3 



Q. 4 
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1.. Nancy uses prints procedures and work orders. Important fully 
understands. *T 

2. Reads print procedures and work orders well. . .follows written 
instructions very well. 

3. throughly uses* asks when not sure 

4. her job is to understand prints and to help others understand them. 



1. has been trained. . .does not discuss it very often 

2. has had training ... not discussed it yet one on one 

3. has had training ... not discussed it yet one on one 

4. trained ... not discueed it yet one on one ~* i 

5. trained ... not discussed it yet one on one . ( 



1. Communicates with other employees. Also speaks native language 
when she can. - ? uj 

2. not shy. ..will speak to everyone when needed m , 

3. comr cates well, 1* am sure will continue to benefit from 
th< jgram 

340 



Question 4 continued. 

1. communicates with everyone : ^ 

2. vorka hard «t co-unic.ting. . . underetanda difficult! can be 

sympathetic - •• 
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1. has participated... sometimes reserved in sneaking her thoughts 

2. has in the past... not active... at the moment - h 

3. has in the past... not active at the moment H 

4. do not remember If she has participated 

5. communicates her ideas on CEDAC process. - < 
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Supervisor Comments Continued 

Supervisor Dan Fleming 



Q. 6 



Employee 1. Same as Q. 5 comment - 

2* Same as Q. 5 comment «-< 

3* Same as Q. S comment ~ ^ & 

4. has participated will speak up in meeting or action center ~ > 

5. will speak her mind - We all value her opinion 5 



Q. 7 



1. works veil in work cell can communicate with everyone ^ > 

1. L is a silent leader , knows and understands what is needed in more than one eel 

3. works well as a teammember ~ S 

4* works well, communicates with others in the work cell z S 

5. works closely with every work cell* offering support - s 



Q. 8 



1. N. has learned many new jobs* This is important to her* ft 

2. has learned new jobs — » smart lady — could cross- train more ~ fj 

3. does work in other ares and communicates well ~ s 
4* has not cross-trained in any new areas / 

5. can communicate well* but a need to improve because of leadership position ~ V 



Q- 9 



1. has participated only a few times, should participate more - </ 

2. has participated, but gets frustrated at lack of support for her ideas 
(lack of employee support) * «^ 

3. has had some ideas implemented* 1 encourage her, hut lacks confidence in her 
English ability h 

4. has not participated to my knowledge ~~ / 

5. has participated, could possibly help implement others' ideas 



Q, 10 



1. has been trained but not made many improvements ~ 7 5 

2. has been trained but not made many improvements * ; 
3* has been trained but not made many improvements * ; 

4. has been trained but not made many improvements v 

5. has made some changes, but could get more actively involved 4 
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Statistical Digest - Supervisor Dan Fleming Continued 

Questions 11,12.13 
Q. 11 

1. B 

2. B 

3. B 

4. B 

5. B and C 

Q. 12 
Comments 

1. continues to improve each year.. .should continue. ..in ESL program 

2. speaks English well, I'm sure she has benefited from the ESL program.. 
I'll encourage her to continue. in the future 

3. communicates veil. I feel program will help her to communicate better 
bet, more important, give her confidence to speak and write more often 

4. has Improved and continues to benefit from the program 

5. important for her to communicate effectively in her leadership role 
She should continue to participate in the program 

Q. 13 

Suggestions for improvement in ESL program 

1. .have guest speakers or guest writers, possibly people they'd be 

2. does student have homework assignments e.g. reading an article in the 
paper or magazine or writing about a subject 

3. give homework assignments every class 

4. CONTINUE TO PUBLISH employees writing - I have enjoyed reading them, . 
have seen improvements, and I know it builds their morale 

5. get more students Involved in writing, possibly examples of work 
related things such as valued ideas forms, minutes, memos 
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Supervisor #1 Questions 1-10 Statistical Digest • Page 1 (P.T.O. Page 2) 

2 3 4 5 6 7 8 9 10 

Employee #1 d d d d+e d d d d d c 

#2ee ceddede c 

#3ee cedceee c 

#4de cdcdedc c 

#5ee d eeea ee d 



0 34-i 
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ST 1 



Total of Respondents : 20 

One was unable to complete because of literacy problems 





Can't Do 


Caa 

But Don't 


Do, Could 
Do Better 


Do Well 


N/A-N/ 


1. Performance Review 






9 


7 


Rest f 




1 

A 


1 

X 


in 


C 


Act) L \ 


3» Understand inc X-Matrix 


2 


1 


11 

A A 






4. Speaking English at work 




2 

mm 


10 




Poof § 


5t CEDAC Group 




1 
* 


15 




pAQf / 


6« Action Centers 


2 


2 


7 


1 
a 


ivcdl \ 


7. Work Cells 




1 


10 


5 


Rest (3 


8. Cross-Training 




M» 


11 


4 


Rest ( 


9. Valued Ideas 


1 




9 


2 


Rest ( 


10. Use of Ergonomics 






6 


9 


Rest < 


Totals ( 1-10 ) 


6 


8 


98 


41 


37 
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ST 2 



11. (1) 

(U) 

(iii) 

12. (i) 
(U) 
(iii) 

13. (i) 
(ii) 
(iii) 
(iv) 

14. (i) 
(ii) 
(iii) 
(iv) 

15. (a) 
(b) 
(c) 
(d) 
(e) 
(f) 
(g) 

16. Comments 

a) 
en) 

(iii) 
(iv) 



Questions 11-20 - Statistical Digest 

17. '\m* - 6 

*4 years ago' - 3 
'1987 1 - 1 
3/91 - 2 

9/90 - 1 

N/R - rest. 

18. Yes - 1 ( , 1989 t - 1 year) 
No - 10 

No response - rest 

19. (Total Months) 
4 years 3 
36 months 2 
24 months 1 
18 months 1 
8 months 1 
2 months 2 

* 

N/R rest . 



G 


M 


P 


4 


9 


4 


1 


12 


4 


2 


10 


5 


2 


7 


8 
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Digest of Comments. (Some slight editing) 



Q. 1 



'•^•v^ A^^ ' r i<S 

1. Before English class, I couldn't do this at all. *%Ji'£M-"1\\*\i* ' 

2. I like to do everything perfect, but I can't because I need more time' 
and more experience. 

3. I think I'm doing my job careful, I do my best. 

4. Sometimes we aften hear CEDAC, X-M, or TE programs. I don't understand 
the whole subject beacuse I'm not too familiar with these words. 

5. 1 understand it well. 



Q. 2 



1. In my job, no opportunities to use blueprints. 

2. Sometimes I need some help, but I do this. 

3. because I can't reading very well 

Q. 3 

1. I can read the X-Matrlx diagram 

2. I try to use English that I learned in the class to discuss X-Matrix. 

3. I need more pratice because I didn't attend the class. 
4.. I can speak, but I can't write about the X-Matrix. 

5. I would like to understand more. 

Q. 4 

1. I use English to any groups whose mother-tongue is not English. 

2. I can communicate very easily about my job because the English classes 
help me too much. 

3. I do my beat to understand to communicate. 

4. I have no problems to communicate with my English speaking friends. 

5. Sometimes I can't understand, but I pay more attention when somebody speaks 
to me. 



Q. 5 



1. I understand, but I can't explain. 

2. At times, I participate actively in CEDAC groups so there in the meeting, 
I use my English. 

3. I need more English language and writing. 

4. I never had the chance to participate in CEDAC. 

5. I would like to participate more. 
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Q. 6 



1. I never participated in one. 

2. Never participated 

3. I never participate. 

4. I never participate. 

5. I sign my name a few weeks ago to participate - never participated 
to writing becuase I cant's speak English well. 



Q.77 



1. No work cell In my area. 

2. 1 think ESL class helps me a lot in my job. 

3. I never had the chance to participate. 

4. I would like to participate, but I don's do this very well. 

5. I like to do better haven't any work cells in my area. 



Q. 8 



1. I never participate 

2. I read and understand, but I can't explain good. 

3. I never had the chance to participate. 
A. Yes, speak English. 

5. English class helps me in cross-training in my job. 



Q. 9 



1. I use ergonomics healthy went on my job (?) 

2. 1 wrote 11 valued ideas so far. I did well. 

3. I need write more Enlgish about valued ideas becasue I like give good 
ideas to help the company group and make more money. 

4. writing in English 

5. I think English class help me for about valued idea. 



Q. 10 



1. I use ergonomics in my job and I nesd more healthy work style. 

2. I learn a lot of health style when I participated in ergonomic program 
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15 Stats 



a. 


10 


use 


English 


with ay children 


b. 


10 


use 


English 


with my friends and neighbors 


c. 


10 


use 


English 


with strangers in market, on street 


d. 


4 


' use 


English 


at my children's school 


e. 


11 


use 


English 


to understand the news 


f. 


9 


use 


English 


in my piece of worship 


g. 


2 


use 


English 


in community activities 



16 - Any other comments 

I like this program very well and I'm very happy to do the same* This year 
I'd read and write a lot. 

I like to continue to do what ve did this year and read more. 

My abilities is 1 use English in my work area, on my job. I feel comfortable. 

1 am happy that now I can use English that 1 learned in class, everywhere - 
to my job or outside, in my own life. I wish that the company and the state will 
continue the English classes to teach the employees who come from other countries. 

Outside the job, I'm more comfortable using English in my family - with neighbors 
with strangers on the street. 

I don f t use English very much outside the job* I can do a very good job 
by talking or communicating with anybody. 

1 speak more English with American people and with friends. 

ESL help me on and off the job. Before this program I can't go alone to the 
doctor. Now I can do it myself and in many other situations. 



Additional Comments 

"... Two hours for grammar and two hours for vocabulary and every student 
must speak the word so that we wll know the way we pronounce wrong or right." 
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Kennedy Die Castings, Inc. 
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CASE STUDY: 
OUTCOME STUDY AT KENNEDY DIE CASTINGS 



Background: Kennedy Die Castings, Inc., Worcester 

Kennedy Die Castings, Inc. is a forty-one year old manufacturer of metal 
castings. It employs about 100 people. Founded by the senior Mr. Kennedy 
when the die-casting business was in its infancy, the company is now run by his 
two sons. Like many small manufacturing companies throughout the state, 
Kennedy Die Castings faces new challenges as it: converts its production 
methods to include sophisticated, computerized machinery, introduces the team 
approach to problem-solving known as "quality circles", and institutes SPC to 
keep costs down and quality up. Management is so committed to making the 
transition from low to high technology that three years ago it invested in 
construction of a sleek new plant designed to make the company leading-edge 
die casters for the next two to three decades. 

No longer able to draw on a labor pool of native English-speaking men from the 
Worcester area trained in die casting techniques by local vocational schools, 
the company is turning to a new labor pool of limited- English-proficient men 
who have not necessarily been trained as die casters. About 65 percent of the 
employees do not have Engflsh as their native language. Most of these 
employees are Southeast Asian, Polish, and Hispanic. Changes in the 
production process demand that these workers adapt and upgrade their skills to 
accommodate increasingly sophisticated machinery and team methods of 
problem-solving. Providing on-site ESL and basic skills instruction is one step 
toward developing a confident, skilled and adaptable workforce. 

The company offers one ESL class, beginner level for 14 students, 4 hours per 
week for two 20-week cycles. Class is located in the company's cafeteria and is 
taught in the afternoon by the teacher/curriculum developer. Employees are 
paid time-and-a-half to attend classes after work. Production demands could 
not tolerate losing people from regular shifts, even if there was long term value 
in attending class. Scheduling classes before/after a shift, and paying 
employees overtime rate solved this problem to everyone's satisfaction. 

Description of the evaluation process: 

The make-up of the evaluation team : The Kennedy team was composed of a 
high-level manager, a shop-floor supervisor; a personnel officer (later replaced 
by a newly-hired human resources officer); one student; and two 
representatives (one the program coordinator, and the other the program 
instructor) of the education-providing agency, Quinsigamond Community 
College. On a day-to-day basis, the work of the project was carried out primarily 
by the two College representatives, with active assistance in the initial stages by 
the personnel officer (who left mid-way through the project due to personnel 
changes in the company). Although the student initially participated actively in 



the team's work, he also left midway through the project when he was injured 
and coukJn'; come to work. 

The process : The Kennedy team followed the ten-step process outlined in 
Section II of the report of the MWEI Cycle 4 Evaluation. As stated above, the 
lion's share of responsibility for the project was left in the hands of the 
Quinsigamond Community College representatives. 

What was teamed about fa education program: See Section n, step 8 of the 

report of the MWEI Cycle 4 Evaluation for a summary of the Kennedy team's 
findings and the attached data summaries. 

What was learned about the evaluation process: Like many other programs 
participating in this project, the Kennedy team had to deal with personnel 
changes, layoffs, and other factors at the company which affected the 
availability of team members themselves and of those from whom data would 
be gathered. This led to a reduced momentum in the project, with key steps 
delayed and a limitation of the amount of data which could be collected. 
Despite these constraints, the team carried through with the work and produced 
the findings and plans for follow-up activity described in Steps 8 and 9 in 
Section II of the report of the MWEI Cycle 4 Evaluation. This persistence was 
due to the dedication of the education staff and to the ongoing support of senior 
company management The senior manager on the team recognized the value 
not only of the education program in general but also that of this evaluation 
project. His interest was evident in his active participation in ail team meetings 
and his ongoing willingness to work with - rather than put unreasonable 
expectations on - the team as it tried to find ways to come up with useful data. 

Attached data-gathering Instruments, summaries of findings, and related 
documents: 

Data-gathering instruments: 

- Interview guide for supervisors 

- Student rating scale 

Summaries of data: 

~ Results of Supervisor Interviews with Student Performance Level 
Rating Scale 

- Tally sheet for student rating scale 

This team did not produce a final report. 
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Kwiifdy Dm Cmiitfi: 
PreusMitary Findings fro* the Supervisor and Student 

Questionnaires 

Prepared by Paul Jurmo, Consultant 
October 1. 1991 

Summary off ina data: 

The raw data for the two uses of the supervisor questionnaire and the 
August use of the student questionnaire are shown in the attached summaries. 

in his two assessments of the students' job-related abiifities, the 
supervisor indicated the following: 

- On average, students fall in the "medium" range of ability. 

- They on average improved slightly during the period of June to August 
in their abilities to hands all of the given job tasks. 

- The students showed the strongest abilities in (a) identifying quality 
defects, (b) functioning as a KDC employee, and (c) comprehending and 
following directions. 

- The students were rated lowest in their abilities to (a) communicate in 
general! and (b) actively participate in KDC training programs. 

- The data could be organized to provide a "ranking" of students. This 
could be used by instructors to identify which students needed more help 
and the skills areas which needed particular attention. 

In their own self-assessments in August, the students tended to rate 
themselves on the high end. They semed particularly confident in their abilities 
to understand their supervisors' erections, tell someone that they don't 
understand something, tell ther supervisors they have a problem, and spot 
defects. 
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Lessons Learned: 



The instructor who gathered the data left that the information was 
instructive in the following ways: 

1. The feedback provided by the supervisor in the interviews tended to 
confirm the teacher's understanding of the students' abilities, but not in 
all cases. The instructor was able to see how a student might perform 
well in the classroom but not be transferring that ability to the worksite. 
This enabled the instructor to clarify where students' skHl gaps were and 
to change the curriculum accordingly. 

2. The process demonstrated to the instructor just how complex her job 
is. At a tin*? when the entire production process is placing greater 
demands on individual workers, she is being asked to customize 
instruction to a number of job-related tasks with which she has no direct 
experience. This has led her to conclude that she needs more technical 
input from the company if she is to effectively tailor the program to the 
company's interests within the limited amount of instructional time 
available. But it is cSfficult to get that input at a time when the 
supervisory-level personnel who might give that input are being replaced 
and also being expected to take on other new responsibilities beyond 
those required by the education program. . 

3. By setting clear goals, the instructor can more effectively monitor 
progress toward those goals. 

Other possible conclusions identified by the consultant: 

1. The process of identifying desired program outcomes and developing 
instruments to measure propjess toward those goals has had the 
following benefits: 



9 

ERIC 



355 



* The goat-setting process has clarified what objectives the 
education program and other company training programs should 
be focusing on. 

* Some useful data have been produced to clarify what impact the 
program is having, skid areas which need more refinement, and 
employees who need particular help. 

* The program now has evaluation instruments which it can further 
refine and expand upon. The team might also develop job-related 
tests (oral or written), simulations, or other means of gathering 
data about student prog-ess. 

* A "team" structure and process has been developed which might 
serve as a foundation for more-effective planning and evaluation 
in the future. 

2. The process of developing and implementing a new evaluation 
"system" has been time consuming and made difficult by the limited time 
which staff could give to the project Staff were also being asked to 
perform tasks (for example, design meaningful data-gathering 
instruments) with which they were unfamiliar. This further slowed down 
the process as staff tried to develop expertise along the way. Fortunately, 
other team members - particularly higher management and the 
supervisor who completed two sets cjj questionnaires - were patient and 
supportive of the process. 

3. Given the promising results of this year's attempt to develop new 
means of measuring program outcomes, program planners should 
carefully consider how they might build on this year's experience to 
design a more-efficient "system." One important needed action: Allocate 
the time and resources needed for at least one team member to (a) 
develop needed evaluation expertise and to (b) then focus on 
coordinating future planning and evaluation activities. 
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4. As students gain new skills, the company needs to be sure that 
students have opportunities and incentives to put those new skills to best 
use. In short, the learners must "use it or lose it." 

5. The program at Kennedy has much to offer others in the workplace 
literacy field. Those involved should be encouraged to share their 
expertise with others trying to develop similar programs elsewhere 
around the state and nation. 
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Kennedy Die Castings: Data from Student Questionnaire. 8/91 
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Quinsigamond Community College Workplace Educate 

Interview Guide For Supervisors- Rating Scale 

1. What is your assessment of OflYAO) * ~ 3~T^ Jb *^ 
communication skills in 0 1 S 

general? 

2. WIYAO ability to function as " ^ ^ H (j*J) ^ 

a KDC employee? 

3t viYAO ability to interact with ^ J ~ 3 *(^) C T 

work peers? 

4. WIYAO ability to recognize and <<-_--- — ~ J & ? 

identify quality defects? ° 

5. WIYAO ability to participate in 

group problem-solving on the floor? ^ f O 4 S <^ 

as a listener 
as an active participant 

as an initiator 

6. WIYAO level of participation 

in the most recent KDC training 

program? 

as listener 0 » i ? i H i 5 ^ 1 




as active participant 
7. WIYAO ability to comprehend 



£ 

o » 



and follow directions? ^ TT ~ T 
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KENNEDY - INTERVIEW GUIDE RATING SCALE 



RESULTS 
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Student Rating Scale 

1. I speak with my coworkers in English. 

a. I don't do it at all. 

b. I can do it a little. 

c. 1 do it very often. 

d. I do it all the time, 

2. I can tell my supervisor I have a problem. 

a. I don't do it at all, 

b. I can do it a little. 

c. I do it very often. 

d. 1 do it all the time. 

3/1 can tell someone that I don't understand. 

a. I don't do it at all. 

b. I can do it a little. 

c. I do it very often. 

d. I do it all the time. 

4. I understand the directions my supervisor gives me. 

a. I don't do it at all. 

b. I can do it a little. 

c. I do it very often. 

d. I do it all the time. 

5. I can find defects and tell what they are. 

a. I don't do it at all. 

b. I can do it a little. 

c. I do it very often. 

d. I do it all the time. 
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6. I participate in group problem- so lying on the floor, 

a. I don't do it at all. 

b. X can do it a little. 

c. I do it very often. 

d. I do it all the time. 

7. I understand what the other people are saying on the floor 
when we are discussing a quality defect. 

a. I don't do it at all. 

b. X can do it a little. 

c. I do it very often . 

d. X do it all the time, 

8. I can tell my supervisor and coworker what the defect is on 
the floor. 

a. I don't do it at all. 

b. I can do it a little. 

c. I do it very often. 

d. I do it all the time. 

9. If I find a defect, I call my supervisor and tell my coworker. 

a. I don't do it at all. 

b. I can di it a little. 

c. I do it very often. 

d. I do it all the time. 

10. I can participate in Kennedy Die Castings, Inc. training sessions, 
and I can undertsand what they are saying. 

a. X don't do it at all. 

b. I can do it a little. 

c. I do it very often. 

d. I do it all the time. 
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If I do not understand something at a training session^ will 
ask a question. 

a. I don't do it at all. 

b. I can do it a little. 

c. I do it very often. 

d. I do it all the time. 



Student Rating Scale - T#L(*9 EST ^/fafcihf'&G 

C*s*'*} r >' :rnc ' 

1. I speak with my coworkers in English. #J &f 

t6% a - 1 don<t d0 it st a11 * \S 

^&*U b. I can do it a little. \/ \f *S 
Jb% c. I do it very often. 

^'/# d. I do it all the time. r\ \S 

2. I can tell my supervisor I have a problem. 
q 9 /* a. I don't do it at all. 

St/ 

C ' 1 d ° ±Z Very often * ^ 
faf'fa d. I do it all the time. S \ \Syf^ 

3. ' 1 can tell romeone that 1 don't understand. 
Q*fa a. I don't do it at all. 

b. I can do it a little. ^ ^ i/ ' 
q0Jq c. I do it very often. 

j£(fft& 1 d0 it: a11 the time - ^SyfSS^zf 

4. I understand the directions my supervisor gives me. 
O*j0 a. I don't do it at all. 

*fl°lo b. I can do it a little. / ^ 

c. I do it very often. ✓ 

5. I can find defects and tell what they are. 
I&*l0 a. I don't do it at all. ^ 

„ . mi c. I do it very often. 
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, I participate in group problem- solving on the flopr, 
a. I don't do it at all. 

b. 1 can do it a little . iv'/X 

MO f* 

c. I do it very often, l/i/^ 

d. I do it all the time. ^ ^yS ^ 

7. I understand what the other people are saying on the floor 
when we are discussing a quality defect, 

0*/6 a. I don't do it at all. 

ijtffa b. I can do it a little. l/^V^C^l^ 

j^jp c - I do it very often, 

jQ?f 9 d. I do it all the time -^|/^y/^/^ 

8. I can tell my supervisor and coworker what the defect is on 
the floor. 

a. I don't do it at all. 

b. I can do it a little. ^ 
£O 0 /q c. I do it very often. 

gp*/ § d. I do it all the time, ^^/y^y/^ 0 ^/ 

9. If I find a defect, I call my supervisor and tell my coworker. 

f($0 a ' 1 don't do it at all. 

/#^b b - 1 can di it: a little. 

£(/{0 c. I do it very often. ^ 

fffo d. I do it all the time. ✓VVC^ / SIS 

10. I can participate in Kennedy Die Castings, Inc. training sessions, 
and I can undertsand what they are saying. 

a. I don't do it at all. . * s 

^0% b - 1 can do il a little. ^ f V * 

£$ 9 I$ c. I do it very often, 

fj^ jj d ' 1 do it all the time. ^ ^^^\/ 
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j.1 . If I do not understand something at a training session I will 
ask a question. 



a. I don't do it at all. 

ffl € U b, 1 can do it a little. iXt^/' 

£Q>*/0 c * I do it very often. 

jff/t <*■ I do it all the time. (/^ 
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CASE STUDY: 
OUTCOME STUDY AT BOSTON CITY HOSPITAL 

Background: Boston City Hospital, Boston 

The main facility of this large urban medical center which serves Boston's poor 
recently underwent major renovations. Old wings were closed, a new one was 
built; staff was trimmed; and the number of beds was reduced- all in the service 
of providing better care to more people. Like other hospitals and health care 
agencies, Boston City Hospital (BCH) faces a critical shortage o tinned 
workers at the entry, aide, technician, and R.N. levels, Basic skills Instiuctton 
and a unique study skills course is one part of the hospital's strategy to retain 
loyal workers and promote them through the ranks. Retaining workers through 
training and promotion is the only way to maintain quality of service in an 
economy where trained health care workers are scarce. 

However the workplace education program does not only recruit students from 
the main 'facility. It recruits workers from three other facilities through union 
membership: the Long Island Hospital on Long Island; the Mattapan Hospital in 
Mattapan, and two nursing homes- Edgewood and Rest Haven. There are 
three unions which support the BCH workplace education program: AFSCME 
1489; SEIU 1475, and SEIU 285. The high degree of union involvement in this 
program distinguishes H. So does the amount of travelling that some students 
do in order to attend their classes Motivation to travel from one end of Boston 
to the other comes In part from the promise of promotion. But transportation is 
often a problem that keeps students from attending classes regularly. 

The BCH workplace education program is unique because it offers two very 
different levels of instruction: the ABE/PRE-ASE/ASE level has one class with 
16 students ranging from beginning literacy through secondary level; the pre- 
technical level has one class with 9 students preparing for post secondary 
health training programs. The pre-technical class is a study skills class which 
prepares people for study beyond the the secondary level. Students who want 
to make the leap to professional training but need to brush up on their study 
skills find a home here. Each class meets twice a week for three hours for a 
total of 6 hours a week in two 16-week semesters. The basic academic skills 
class is open entry/exit. Classes are located in a classroom in the hospital's 
training wing. Both classes are taught by the teacher/counselor/coordinator. 

Tutoring and individualized career counseling time for students in both classes 
is also available by appointment in the mornings and on Friday afternoon in the 
teacher/counselor's office. 

Description of the evaluation process: 

Thfl make-up of tha evaluation team: The BCH team suffered serious losses 
during the course of the project, to the point where it was not clear that a "team 
in fact existed by the end of the project. The team originally consisted of the 
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program coordinator/instructor, a representative of the hospital's health 
education center, a union representative, two supervisory-level nurses, and a 
nursing assistant who was a student in the program. Subsequently, the 
coordinator left his job, the health education representative went on maternity 
leave, and one of the nurse supervisor's was transferred, and the student 
stopped participating in the team when her supervisor stopped coming. In the 
last team meeting, only the union representative and one of the supervisory 
nurses attended, although they showed dear interest and ability to carry on the 
work of the team in some form. 

The process : The BCH team followed the ten step process described in Section 
II of the report of the MWEI Cycle 4 Evaluation. However, the pace of these 
steps was not steady due in large pan to the attrition described above. 

What was teamed about the education program: See Section II, Step 8 of the 
report of the MWEI Cycle 4 Evaluation for a summary of the BCH team's 
findings. 

What was teamed about the evaluation process: The team's progress was 
limited by the attrition of its members as well as by upheavals going on in the 
hospital at the time of the project. Staff cutbacks and schedule changes meant 
that not only team members themselves but the students and supervisors the 
evaluation should have been relying on, were often not available. The loss of 
the program coordinator midway through the project meant that the vital 
continuity, contacts, and records which he maintained were no longer under 
anyone's direct control. This made it that much more difficult to communicate 
with the students and supervisors who were to serve as informants in the study. 
And, even before he left midway into the project, the project coordinator was 
winding down his involvement in the outcome study, as It was of lower priority to 
him as he attended to the more-immediate tasks required to wrap up his work in 
the education program. One other limitation was the fact that there was no 
ongoing involvement in the evaluation team of high-level management (A high 
level manager came to the January 10th training but not to subsequent on-site 
meetings.) This made it difficult for team members to be clear about how the 
evaluation - and the education program itself - fit into the hospital's plans for 
human resource development. Despite all these limitations, the team did make 
steps toward clarifying the outcomes which the education program should be 
achieving. The team hoped to reformulate itself -- this time with high-level 
management support - and incorporate these objectives into future education 
services at the hospital. 

Attached data-gathering instruments, summaries of findings, and related 
documents: 

Data-gathering instruments: 

- Questionnaire for supervisors 

- Questionnaire for students 
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Summary of findings: 

This team did not produce a final report per se. See Section II, Step 8 of 
the report of the MWEI Cycle 4 Evaluation for a summary of the BCH team's 
findings. 
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QUESTIONNAIRE 
TO BE COMPLETED BY EMPLOYEE 
PARTICIPATING IN EDUCATION PROGRAM 

AT 

BOSTON CITY HOSPITAL 

5/29/91 DRAFT 



Name of Employee 

Dale questionnaire completed 
Signature of Employee 



BCH would like to know how successful the education program has been in helping you 
on and off your job. Could you please fin out this questionnaire, so we can know how to make the 
program serve you better? Your responses wiO be kept confidential. 

Please rate your ability to carry out the following duties: 



Staff maetlnge: 

1. In staff meetings, I understand the topics being discussed and respond when asked questions. 

Doesn't apply. 

I cant do this, 

I do this better than I did a year ago. 

I have always been able to do this well. 

Explain your answer: 



2. In staff meetings, I speak up with ideas and information of my own. 

Doesn't apply. 

I cant do this, 

I do this better than I did a year ago. 

I have always been able to do tiiis well. 

Explain your answer: 



3. In staff meetings, I give suggestions about how to improve the operation of my department. 

Doesn't apply. 

I cant do this. 

I do mis better than I did a year ago. 

I have always been able to do mis well. 

Explain your answer: 



1 



Writing on the) Job 



4. When I am asked to out farms (for example, patient charts, inventory forms, or lab 
requisitions) I write the required information. 

Doesn't apply. 

I cant do this. 

I do this better than I did a year ago. 

I have always been able to do this we8. 

Explain your answer; 



5. When I am asked to fSI out forms (for example, patient charts, inventory forms, or lab 
requisitions) I write in clear and correct English. 

Doesn't apply. 

I cant do this. 

I do mis better than I did a year ago. 

I have always been able to do this well. 

Explain your answer: 



Communicating under prsaaure: 

6. When I have to deal with an emergency, I express myself clearly and constructively. 

Doesnt apply. 

I cant do this. 

I do mis better than I did a year ago. 

I have always been able to do thie well. 

Explain your answer: 



7. When I have to deal with a conflict, I express myself clearly and constructively. 

Doesnt apply. 

* can't do this. 

i do this baler than I did a year ago. 

I have always been able to do mis veil. 

Explsin your answer: 
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CoRnnuntesfliig with m panrts w i : 

B. When communicating with my supervisors, I express myself dearly and constructively. 

Doesn't apply. 

I cant do this. 

I do this better than I did s year ago. 

1 haw always been able to do this well. 

Explain your answer: 



Providing high-quality aarvfea: 

9. I do high-quality work, 

Doesnl apply. 

I canl do this. 

1 do this better than I did a year ago. 

I have always been able to do this well. 

Explain your answer: 



10. I anticipate special needs which come up on my job (for example, for a wheelchair) and make 
sure thst these needs are taken care of. 

Doesn't apply. 

I cant do this. 

I do this batter than I did a year ago. 

I have always been able to do this well. 

Explain your answer: 



11. Whan I communicate with others (for example, patients or staff members), I speak clearly, 
politely, and helpfufly. 

Doesn't apply. 

I cant do this. 

I do this better than I did a year ago. 

I have always been able to do this weB. 

Explain your answer: 



12. 1 know my job thoroughly and do not need to be given specific orders. 

Doesn't apply. 

I cant do this. 
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I do this better than I did a year ago. 
I have always bean able to do this watt. 



Explain your answer: 



Allying fha education program to ny Job: 

13. 1 talk about the classes with other people at work 

I dont do this. 

I have dona this a few times. 

I do thff *gutariy. 



Explain your answer: 



14. 1 do more reading , writing, and math on my job. 

I donl do this. 

I have done mis a few times. 

I do this regularly. 

Explain your a.iswer: 



15. 1 read my union bulletins. 

Doesn't apply. 

I cant do this. 

I do this better than I did a year ago. 

I have always been able to do this well. 

Explain your answer: 



16. If I were asked to write an article for the union bulletin: 

I wouldn't know how to do it. 

I could do ft, but would be reluctant to do it. 

I would do it. 



Explain your answer: 



17. If I were asked to speak about the classes at a union meeting or to another group: 
I wouldnl know how to do it. 
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I could do ft, but would bo reluctant to do H. 
I would do it. 



Explain your answer: 



Interact In hospital work: 



18. 1 am interested in working m other jobs in hospitals, and 1 look into job opportunities in this 
field. 

Doesnl apply. 

I cant do this. 

I do this better than I did a year ago. 

I have always been able to do mis weU. 

Explain your answer: 



fcfy rlfihto on tfia Job: 

19. 1 understand my rights on the job (for example, discrimination, pay rates, benefits) and make 
certain that they are respected. 

Doesnl apply. 

I cant do mis. 

I do this better than I did a year ago. 

I have always been able to do this well. 

Explain your answer: _ 



Further oduestlon. training, ond Job opportunltlto: 

20. Please check all that apply to you. 

I have recently applied for a GEO program. 

I have recently applied for the Pre-Tech Progrem. 

I have recently applied for college. 

I have recently applied for a technical training program. 

I have recently applied fa a different job. 

Explain your answer: 
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Off the Job: 



21. When I'm not at work, I talk to other people about the education program. 

Doesn't apply. 

I cant do this. 

I do this better than I cHd a year ago. 

< haw always been able to do mis well. 

Explain your answer: 



22 When I'm not at work. I use more resting, writing, and math in my daily He (for example, getting 
a loan, applying for a job). 

Doesn't apply. 

I cant do this. 

I do this better than I did a year ago. 

I have always been able to do this well. 



Explnh your answer: 



23. (For those employees with school-age children): When I'm at home, I help my children with 
their homework. 

Doesn't apply. 

I cant do this. 

I do this better than I (fid a year ago. 

I have always been able to do this well. 

Explain your answer: . 



24. When I'm at home, I do my own homework with the help of family members or friends. 

Doesn't apply. 

I canl do this. 

I do this better than I did a year ago. 

I have always been able to do this well. 

Explain your answer: 



Additions! eommestte: 
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25. Please odd any other comments, information, concerns, or examples about your abilities on 
off the job which you think we should know about: 



Thank you for your cooperation in fifing out this questionnaire. 
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QUESTIONNAIRE 
TO BE COMPLETED BY 
SUPERVISOR 
ABOUT 

EMPLOYEE PARTICIPATING IN EDUCATION PROGRAM 

AT 

BOSTON CITY HOSPITAL 

5/29/91 DRAFT 



Name of Supervisor 

Name of Employee 

Date questionnaire completed 
Signature of Supervisor 



BCH would Ike to know what impact the education program has had on employees' job 
performance and personal development Because you are familiar with the above employee's 
work, we ask that you pfeese fiU out this questionnaire. Results wfll be kept confidential. 

Please rate tire above employee's abilities to perform the following job-related duties: 
Participation In etcfl msetlngo: 

1. In staff m eatings, the employee responds to the topic being discussed by dearly stating 
relevant information and ideas. 

Doesn't apply. 

Cant do this. 

Can do this better than one year ago. 

Has always done mis well. 



2. In staff meetings, the employee takes the initiative by raising useful issues and volunteering 
information. 

Doesn't apply. 

Cant do this. 

Can do this better than one year ago. 

Has always done mis well. 



3. In staff meetings, the employee gives concrete suggestions about how to improve department 
operations. 

Doesn't apply. 

Cant do this. 

Can do this better than one year ago. 

Has always done this well. 

Use* of documents: 

4. When filling out records (for example, patient charts, inventory forms, lab requisitions) the 
employee provides accurate and appropriate information. 
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Doeanl apply. 

Csnl do this. 

Can do this bettor than one yoar sgo. 

Has always dans this wet). 

6. When filling out documents (for example, patient charts, lab requisitions, inventory forms) the 
employee uses dear, pjsmmsticaJy-corrsct Engfish. 

Doesn't apply. 

Canl do this. 

Can do this better than one year ago. 

Has always done this weB. 

Communicating In stressful situations: 

6. *Vhsn dsafing with stressful situations (e.g., emergencies), the employee expresses 
her/himself clearly and constructively. 

Doesn't apply. 

Canl do this. 

Can do this better than ens year sgo. 

Has always done mis wsi. 

7. When dealing with emotionally-charged conflicts, the employee expresses her/himself clearly 
and constructively. 

Doesn't apply. 

Canl do this. 

Can do this better than one year sgo. 

Has always done this wsfl. 

Communicating with supervisees: 

8. When communicating with supervisors, tile employee communicates clearly and 
constructively. 

Doesn't apply. 

Canl do this. 

Can do this better than one year ago. 

Has always done this wefl. 

Employee's Impact en quality of deportment's car vices: 

9. The employee generally does high-quality work. 

Doesn't apply. 

Canl do this. 

Can do this better than one year ago. 

Hss always done this well. 

10. The err, >yee anticipates special needs (e.g., a wheelchair) and iakes the initiative to see that 
they are taken care of. 

Doesn't apply. 

CsjvI do this. 

Can do this better than one year ago. 

Has always done this well. 
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In a praftssionat mannar. speaking deeriy, pofitely, and heJpfiiy. '* ^ acts 

Doesnl apply. 

Cant do this. 

Can do tNa bettor than one year ago. 

Haa always dona this wefl. 

12. Tha employes is thoroughly knows his/her job and doas not naad constant supervision. 

Doesn't apply. 

Cant do this. 



Can do this batter than one year ago 
Has always done this well. 

Mating fins •titration program to tha Jab: 

^m^SZZ^^ and 8kSta m *• W«" H the 

Doesnl apply. 

Canl do this. 

Can do this batter than one year ago. 

Has always dona this wefl. 

Intaraat in maticaJ wars: 

14. The employee demonstrates curiosity and interest about her/his current job. 

Doesnl apply. 

Cant do this. 

Can do this better than one year ago. 

Has always done this wed. 



15. The employee demonstrates an interest In pursuing job opportunities within the medical field. 

Doesn't apply. 

Canl do this. 

Can do this better than one year ago. 

Has always done this well. 



Additional comments: 

16. Overall, how would you rate the employee's ability to hsnde the above tasks during the past 
month? 

Has shown no significant Improvement. (If you check this one, skip Question 17.) 

Has shown some small improvement. 

Haa shown great improvement. 

17. If the employee's performance has improved, to what do you attribute the improvement? 
(Check an that apply, ranking them in order of impact on the employee, with "1" having the most 
impact) 

Has learned from on-the-job experience. 

The education program has helped. 
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Has learned from other experience. (Please specify:) 



18. Please add any other comments, information, concerns, or examples about this staff 
member's job-related thinking and communication skills which you think we should know about: 



Thank you for your cooperation in fifing out this questionnaire. 



Appendix B: Evaluating the Outcome Evaluation 
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EVALUATING THE EVALUATION: 
ASSESSING THE OUTCOME STUDY 
CONDUCTED FOR THE 
MASSACHUSETTS WORKPLACE EDUCATION INITIATIVE, 1991 



As a participant in this year's outcome study (conducted by either Laura Sperazi 
or Paul Jurmo), please complete the following questions. This will help the 
Initiative to determine what role evaluation should play in future workplace 
education programs. 

Name 



1 . My team worked with (circle one) Laura Sperazi/Paul Jurmo. 

2. 1 understood the goals of this outcome study project to be: (please describe 
the goals): 



3. How well did the consultant explain these goals to your team? 



4. How fully did your team achieve the above goals? 



5. Did the outcome study achieve any additional results (either positive or 
negative)? (Please describe.) 



6. Was the outcome study process useful? (Explain your answer) 



7. Did your team gain new knowledge or skills as a result of going through the 
outcome study process? (Explain your answir.) 



8. What might the MWE1 do in the future to improve the assessment of 
outcomes of your workplace education program? Please tell us how useful you 
think the following activities might be, and then add your own comments. 

• Offer training in how to computerize data at the beginning of the outcome 
evaluation period. 



• After Phase I (identifying outcomes) of the outcome evaluation is completed, 
circulate a grid to all the evaluation teams of the various outcomes which 
programs identified, so that teams can compare and contrast their outcomes 
with those of other teams 



• After Phase I (identifying outcomes) of the outcome evaluation is completed, 
convene a meeting of representatives from the various evaluation teams so that 
teams can compare and contrast their outcomes with those of other teams. 



• Build the time required to conduct a meaningful evaluation more clearly into 
the MWEI RFP so that education staff, company representatives and union 
representatives are prepared for the work they are asked to do. 



• Convene a workshop or seminar day on outcome evaluation which is separate 
from the work done on-site, and which interested members of the evaluation 
teams could attend- a "thinking through the issues" workshop. 



• Provide exercises/instruction in team building for the evaluation teams which 
would help the group to work together a group. 



• Other things the MWEI might do in the future to help you assess the outcomes 
of your employee education program? 



9. Please list any other comments, suggestions, ideas you might ha^e related 
to outcome evaluation. 



Thank you for your help. Please return this questionnaire by September 3, 
1991 to: 

Laura Sperazl 
Evaluation Research 
1589 Centre Street 
Newton Highlands, MA 02160 
617/527-6081 



Appendix C: Curriculum Study Questionnaires and Related 

Documents 



References 



"Introduction to Kennedy Die Costings, Inc. Curriculum", o three-page 
introduction written by program staff, Is available from the Workplace 
Education Program coordinator at Quinsigamond Community College 

Curriculum Deueloomant at the South Coue Manor Nursing Home by 
Marianne Caldwell, Hsiao Chang and Johon Uvln (Draft Manuscript) 1991 

"English at Work - fl Toolkit for Teachers", by Deborah Dorndt, I* jry 
Ellen Bolfiore and Jean Handscombe. New Readers Press, 1991 



Resources for Thinking/Study Skills 

Learning to Learn" course at Roxbury Community College 

Methods of inquiry by Marcle Heimen and Joshua Slomlanko, Learning 
to Learn, Inc. 28 Penniman Rd., Rllston., MR 02134 1988 

Critical Thinking Skills fo r the Duality Workforce Learning to Learn, Inc. 
1991 



WORKPLACE EDUCATION INITIRTIUE CURRICULUM STRNDRRDS 



1. PHILOSOPHV/RPPRORCH RNO GCRL5 

1.1 There is a curriculum/Instructional 
philosophy/approach. 

1 .2 The program's educational goals are clear to 
teachers, students, and the evaluation team. 

It is clear to students, teachers, and others 
where the curriculum Is supposed to help students 
end up. 

2. CURRICULUM 

2.1 There Is a curriculum development model. 

2.2 The curriculum development model is o process 
of helping student/workers (as a group or indiv- 
idually) articulate their needs, goals and con- 
cerns and addressing these through instruction. 

2.2.1 This process Is described In print. 

2.2.2 Teachers, students, and the 
evaluation teom ore oil oble to 
describe this process clearly. 

2.2.3 This process Is ongoing. 

2.2.4 It goes bock and forth between the 
written design, actual 
implementation, and revision of the 
design. 

2.3 Curriculum is toilored to the worksite and to 
students' "life skills 1 ' (and work) needs. 

2.4 Teachers, students and partners (company, 
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union, education provider) ore Involved In 
curriculum development. 

2.4.1 Teachers ere Involved In curriculum 
development. 

2.4.1 .1 Teachers are peld for fteir 
involvement. 

2.4.2 students ere Involved In 
determining curriculum objectives. 

2.4.3 Partners revieiv curriculum regularly. 

The curriculum is related to the work 
context. It uses this context for students' 
learning objectives. 

2.5. ! Student/workers' language/grammar 
needs at work. 

2.5.2 Student/workers' other basic 
skills needs at work. 

2.5.3 Work-related Issues. 

2.5.4 Union/labor issues/concerns. 

2.5.5 Student/worker empowerment. 



The curriculum is releted to students' lives 
outside work. It uses this as the context for 
students' learning objectives. 

2.6.1 Basic skills needs such as helping 
children with school work, writing 
notes to a child's teacher, reading 
to children, reeding the Bible, 
doing Income tax, etc.) 



2.6.2 


Language/grammar needs outside work 


2.6.3 


Life skills 


2.6.4 


Individual growth 


2.6.5 


Family Issues 


2.6.6 


Community Issues 



2.7 The curriculum addresses some or all of the 
following skills: 



2.7.1 


Listening 


2.7.2 


Speaking 


2.7.3 


Reading 


2.7.4 


Minting 


2.7.5 


Computation 


2.7.6 


Interpreting skills 


2.7.7 


Life skills/survival skills 


2.7.8 


Teamwork/cooperation 


2.7.9 


ability to work independently 


2.7.10 


Critical thinking 


2.7.11 


Problem solving 


2.7.12 


Learning about others' cultures 


2.7.13 


Management skills 


2.7.14 


Secandary subject erees 


2.7.15 


Grammar 



2.8 The curriculum addresses the Issue of 
different levels of skills in the classroom. 

2.9 The curriculum addresses the student/worker's 
understanding of the company as a whole, and how 
its parts Inter-relate, Its operations, and how the 
student/worker's part reletes to the whole. 

INSTRUCTIONAL METHODS RNO RCTIUITIES 

3.1 There is a variety of methods and activities. 

3.2 ESL literocy end lenguege practices ere 
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Integrated in the classroom. 

3.3 IN ESI dosses, reel life language Is encouraged. 

3.4 Teaching and learning approaches are centered 
around key events and problems, along with 
everyday activities In students' uiork, homes or 
community environments. 

3.5 Instruction addresses student/workers' 
understanding of the company as a whole end how 
its parts inter-relate, Its operation, end how the 
student/worker's pert relates to the whole. 

3.6 Student/workers hove a choice over learning 
activities. 

3.7 Student/workers are not mostly filling out 
workbook peges. 

3.8 Methods address student/worker empowerment. 

3.9 Methods are tailored to the worksite and to 
students' "life skills" (ond work) needs. 

INSTRUCTIONAL M ATE B IRIS 

4.1 There ore some teacher-mode materials. 

4.2 Instructional materials have been eveluoted by 
teachers (end others) for 



4.3 Some of the instructional materials are tactile . 

4.4 Some of the instructional materials are non-print. 

4.5 Student/workers hove e choice over materials they 



4.2.1 
4.2.2 
4.2.3 



Being based on adult subject matter 
Ethnicity Issues 
Gender issues 



use. 

4.6 Materials are based an adult subject matter. 

PBOGRRM DESIGN ISSUES 

5.1 Class teacher/student ratios are appropriate. 

5.2 There are sufficient instructional hours/week. 
5.5 There Is on I.E.P. 

5.5.1 It Includes student learning 
objectives. 

5.5.2 It Is derived, in part, from the 
inltiel assessment. 

5.5.5 It Includes Information about stu- 
dents' preferred learning styles. 

5.4 Instruction is self-paced, following the objectives 
In the I.E.P. 

5.5 Some Instruction Is group-paced. 

5.6 There are Incentives for students to make 
progress. 

5.6.1 There are extrinsic Incentives 
such as higher wages, bonuses and 
promotions. 

5.6.2 There ore Intrinsic incentives 
such os public recognition of 
achievement, appreciation, and 
increased responsibilities. 

5.7 Learning differences/disabilities are identified 
and addressed In the program. 
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5.8 



Increasing student/workers' self confidence Is 

address ed In the education program. 



5.9 (For ABE programs) the Issue of economic class, 
and Its relationship to self confidence and 
learning progress Is addressed. 

5.10 Teachers' class preparation time Is peid for. 

5.1 1 There ore regular times for stoff meetings. 



5. 1 2 Stoff ore Involved In educetionol program design. 

5. 1 3 Program Coordinator teaches. 

5.1 4 Program Coordlnotor does curriculum development. 

5.1 5 Relotlontfilps with the learning provider, compeny, 
and union one supportive. 

6. STUFF ORIENTATION AND STUFF DEVELOPMENT 
6.1 New teachers attend orientation. 



5.11.1 



These Include time for curriculum 
development work. 



6.1.1 



Orientation Includes familiarizing 
them with the work objectives, 
routines, volues concerns, 
personnel,ond terms of the compeny. 



6.1.2 



It includes familiarizing them 
with the objectives, structure, 
personnel, terms ond concerns of 
the union. 



6.2 Stoff attend ongoing training ond stoff 
development activities. 



6.2.1 Stoff ore paid for staff development 
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time. 



6.2.2 Staff development is done In 
relation to curriculum development. 

6.2.3 Staff development Is related to 
teacher evaluation. 

6.2.4 It includes cultural sensitivity 
and awareness. 

6.3 Staff network/coordinate with people In other 
workplace education programs and other literacy 
programs. 

7. IMPnCT OF THE CURRICULUM ON WORK, THE COMPANY UNO UNION 

7.1 The contents of the curriculum hove en Impact on 
the company or the union or company or union 
culture (practices, policies, and/or priorities). 

7.2 Language use In the classroom Is related to 
language used In work. 

7.2.1 Work-related language needs are 
brought into the classroom 
curriculum and instruction. 

7.2.2 Language (vocabulary, syntax, 
pronunciation, etc.) learned in the 
classroom Is used In the workplace. 

8. INSTRUCTIONAL SETTING 

8.1 The physical space Is: 

8.1.1 comfortable 

0.1. 2 Reasonably sized 

8.1 .3 Well-lighted 

8.1.4 Cheerful 

8.1 .5 R place which establishes a sense 
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of belonging 



The furniture Is: 



8.2.1 odult-slzed 

8.2.2 arranged comfortably 
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PROTOCOLS 




DOCUMENT REUIEUJ CHECKLIST 



DOCUMENTS RURILRBLE? RECEIUED? COMPLETED? 

1. Curriculum 

2. Instructional materials 

3. Assessment Instruments 

4. Staff orient, materials 

5. Individual Learning Plan 

6. Program Drochure(s) 

7. Others(t) 
Bajdeju of fact shppt 

BejjjejU of brochures and program products 
Bfliltelii of curriculum 
Beldam of Instructional materlols 
Bauieju of Assessments 

BfliiAflUi of Staff Orientation Materlols 

Reuleui of I.E-P. 
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Review of...... 



PH i LOSOPH WRPPRORCH AND SOULS 



o What Is the education program's philosophy or Instructional 

approach? (CURRICULUM, TEACHER ORIENTATION MATERIALS) 

o What are its goals? Ulhere Is the curriculum supposed to help 
students end up? 

I. CURRICULUM 

o What is Its model, especially with regard to; 

o curriculum development, 

(Do program staff view curriculum development as a process of helping 
student/workers (as a group ond/or Individually) articulate their 
interests, needs, goals, and concerns and addressing these? If so, is 
the process described in print? 

o is there evidence that the curriculum is tailored to the worksite and 
work needs? 

o Is there evidence that the curriculum is tailored to meet students' 
life skills needs? 

Sources of Curriculum Goals 

o To what degree does the curriculum address student/worker 
needs/objectives which grow out of: 

o Language/grammar needs at work 

o Other bosic skills needs at work 

o Issues at work 

o Union/Labor Issues 
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o Student/worker empowerment 



o Basic Skills Needs In Worker/student*' Liues 
Outside Work (Children's' schoolwork, writing notes 
to o child's teacher, reading to children, reading 
the Bible, doing income tax, etc.) 

o Language/grammar needs at work or outside work 

o Life skills 

o Individual growth 

o Family issues 

o Communltg Issues 



o What skills are addressed In the curriculum: 

Listening? speaking? reading? writing? computation? 
Interpreting skills? life skills/survival skills 
Teom-work/co-operotlon? Ability to work Independently? 
Critical thinking? Problem-solving? (What does this mean in this 
context? Task analysis? Gluing instructions orally and in writing? 
Something else?) How to learn about others' cultures? 
Management skills? Others? Secondary level subject areas? 
(which ones?) Grammar 

o Does the curriculum address the issue of different levels of skills In 
the classroom? If so, how? 

o Does the curriculum address the student/workers' understanding of 
the company as a whole and how its ports inter-relate, its operations, 
and how the student/worker's port relates to the whole? 

o What oppeer to be the major areas of Innovation of the curriculum? 

o What appear to be the major proh ems or limitations of the 
curriculum? 



2, INSTRUCTIONAL MATERIALS 



o lire there teacher-made materials? 

o Are the materials based on adult subject matter? 

o Are the materials sensitiue to issues of :ethnlclty? 
gender? 

o Are some of the materials tactile? 
o Are some non-print? 
3. 1.E.P. 

o Is there is an I.E.P. Does it include student learning objectives? Is it 
directly related to the Initial assessment? Ooes it include information 
about the student's learning styles?) 
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PROGRAM COORDINATOR INTERVIEW PROTOCOL 



COORDINATOR BRCK6R0UND 

o What kinds of work responsibilities do you have? 
Administration? Teaching? Curriculum Oeuelopment? Others? 

PHILOSOPHY, GOALS, MODEL, RPPRORCH 

o What Is your education program's philosophy or 
Instructional approach? 

o What are Its gools? "Where Is the curriculum supposed to 
help students end up?" 

o What is Its model, especially with regard to: 

o curriculum deuelopment; 

Do program staff uleiv curriculum deuelopment as a 
process of helping student/workers (as a group 
and/or Individually) articulate their Interests, 
needs, gools, and concerns and addressing these? 
If so, is the process described In print? Can the 
teacherts) describe the process? Can others? 
Who? 

o teaching; 

o selection of materials and methods; 

o student involvement in decision-making (if this is 
not answered by the question "What Is its 
philosophy?") 

o What do student/workers need to do in the program in 
order to get something from it? 

CURRICULUM RND CURRICULUM OEUELOPMENT PROCESS 
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Hour was the curriculum created? 



Which people haue been inuolued? In what ways? find 
to what degree? Teachers? Education Coordinator? 
Employer? Union Dep.? Supervisors? Student/workers? 
Others? 

o Is curriculum deuelopment on ongoing process that goes 
back and forth between the written design and the actual 
implementation and revision of the design? 

If so, Who is involved in the process of curriculum 
development/reulsion now? 

Now much time each week/month is spent on this by each of 
those Inuolued? It this paid time? 

Do the partners review the curriculum? IDhen? 

Sources of Curriculum Goals 

o To whot degree does your program use as the context for 
students' learning objectives: 

o Language/grammar needs at work 

o Other basic skills needs at work 

o Issues at work 

o Union/labor Issues/concerns 

o Student/worker empowerment 

o Basic skills needs In worker/students' Hues 
outside work (children's* schoolwork, writing notes 
to a child's teacher, reading to children, reeding 
the Bible, doing Income tax, etc.) 

o Language/grammar needs outside work 
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o Life skills 



o Individual growth 
o Family issues 
o Community issues 
o Other? 

o To whet degree do student/workers neve choice over 
curriculum objectives? 

o If the curriculum Is related to the work context, how 
exactly is this relating done? (e.g. did/does the 
teacher go to the shop floor and collect reolle? Other 
ways?) 

o What skills are addressed in the curriculum: 

Listening? speaking? reading? writing? grammar? 
computation? Interpreting skills? life skills/survival 
skills? Team-work/co-operation? Ability to work 
Independently? Critical thinking? Problem-solving? 
(Whot does this mean In this context? Task onolysis? 
6ivlng instructions orally end in writing? Something 
else?) How to learn about others' cultures? Management 
skills? Secondary level subject areas? (which ones? 
Others? 

o Does the curriculum address the Issue of different levels of skills in 
the classroom? If so, how? 

o How Is the curriculum organized? IF III, Is the curriculum 
grammar-based, skills-based, organized by life skills or work skills 
competencies? 

o Soes the curriculum address the student/workers' understanding of 
the company as a whole and how its ports inter-relate, its operations, 
and how the student/worker's part relates to the whole? 
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o What do you consider the major areas of Innouotlon In your 
curriculum? 

o What do you consider the major problems or limitations of your 
curriculum? 

INSTRUCTIONAL M ETHODS AND RCTIUiTIES 

o What methods, activities, practices, and/or 
teaching/learning strategies are used? 
Presentation/lecture? Discussion? Drill ond practice? 
Case studies? Co-operative activities? Dialog 
Journals? Jazz chants? Picture stories? Use of videos? 
Uocobulary developed from on-site responsibilities? 
Reading comprehension derived from family stories? 
Slmulotlon/role-ploys? CHI? Modelling skills? Others? 

o Of ESL), are language and literacy practices Integrated 
in the classroom? Describe. 

o (If ESL), is "real life language" encouraged? Describe. 

o Rre the methods and activities tailored to the worksite 
and ivork needs? 

o Rre the methods ond activities tailored to students' life 
skills needs? 

o To what degree do student/workers have choice over 
Learning strategies/activities? 

o Now are students grouped? Dy objectives? Dy culture ond 
language? Dy educational level/ability? Dy immediate 
work partners? 

o Does Instruction address the student/workers' 
understanding of the company os a whole ond how Its parts 
inter-relate, its operations, and how the 
student/worker's port relates to the whole? 

INSTRUCTIONAL MATERIALS 
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o What meteriols are used? 

Print and non-print? Pictures? Tactile? 

o lllhere do they come from? Teacher-made? Commercial? 
Both? 

o Horn are they eualuated? Ufho evaluates them? 

Are the materials screened to make sure they are bated on adult 
subject matter? 

Are materials screened for ethnicity issues? 

Are materials screened for gender issues? 

Are materials screened for other purposes? 

o To whet degree do student/workers heue choice over materials the 
cless uses? 

o To what degree do student/workers heue choice over materials 
they, as Individual learners, use? 

PROGRAM mm ISSUES 

o Does the program use I.E.P.S? If so, do they Include 
student learning objectives? Are they derived, in pert, 
from the Initial assessment? Do they Include information 
ebout the student's preferred learning styles? 

o To ivhet degree is the program self-poced, following the objectives 
In on I.E.P., and to whet extent group-paced? 

o Are there specific, structured incentives for student learning 
progress; higher wages, bonuses, promotions, public recognition of 
achievement, appreciation, increesed responsibilities, or others) If so, 
what are they? 

o Does the program address the Issue of preferred learning styles ( 
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colloboretlve, independent, dependent)? If so, how? 

o Does the program address learning differences/disabilities? If so, 
how? 

o Does the curriculum or Instruction oddress the Issue of increasing 
student/workers' self confidence? If so, how? 



o Is staff preparation time paid for? 

o Are there regular times set aside for stoff meetings? For 
what purposes? Collaborate to write curriculum? 
Educationol Program Design? 

o Ore there other wags In which staff collaborate to write curriculum 
or meet to review program design? 

STAFF ORIENTATION AND STAFF DEUELQPMENT 

o How ore teachers oriented to the program? 

o Does the orientation Include familiarizing them with the work 
objectlues, personnel, structure, routines, values and terms of the 
business/industry (company and union) involved In the partnership? 

o What paid staff development Is there? Is It related to 
curriculum development? Is It structured? How? Is it 
connected to teacher evaluation? If so, how? Does It 
include cultural sensitivity/awareness? 

o Are there paid staff development opportunities outside the program 
(state and national conferences?) If so, give examples. 

o Do staff in the program network/coordinate with other workplace 
education programs? Other literacy programs? If so, give examples. 

IMPACT OF THE CURRICULUM ON WORK, COMPANY, AND UNION 

o Is there evidence that what is included in the curriculum hes impact 
on the compony or union; on compony or union culture, i.e. practices, 
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policies, or priorities? If so, describe. 

o How Is longuoge use outside the classroom (on the "shop floor* ) 
connected with longuoge used In the clossroom? 

TEACHERS' QUALIFICATIONS 

o What knowledge, skills end experiences ore particularly useful for 
workplace ed. teachers to have? 

PROGRAM PRODUCTS 

o Are there any program products? Published student writing? 
Newsletters? Uideotopes? Community-related projects?) If so, what? 

INFORMING OTHERS RBDUT THE PROGRAM 

o How do you keep people Informed about what Is going on the 
program? 



\ t . a. 
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CLASSROOM OBSERVATION 



o Clots tlze: How many students ore present? How many teachers? Is 
teocher-student ratio appropriate to type ond level of services? 

o Is physical setting appropriate (enough space, privacy, appropriate 
furniture, comfortable temperature, enough light, "learner-centered", 
cheerful) ? 

o Oo methods and materials match adults' learning styles? 

o How ore ESL literacy and longuoge practices Integrated? 
Is real-life language encouraged? or do students mostly fill out 
workbook peges? 
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STUDENT /WORKERS INTERUIEUJ 

o Are classes provided et convenient times end pieces? 

o Are you treeted.es e full partner* in the educetion process? 

o What are the curriculum's goals? Where is the curriculum 
supposed to help students end up? 

o Does your curriculum include: 

o Language and/or other bad e skills needs at mnrk 

o Issues at mark 

o Basic Skills Needs in Worfcar/iturianti' I inpt 
Outside tiinrfc (Children's' schoolwork, writing 
notes to a child's teacher, reading to children, 
reeding the Bible, doing income tex, etc.) 

o Union/Labor issues 

o Language/grammar needs et work or outside work 
o "Life skills" 

o Other? 

o To what degree ere students involved in decision-making 
about the curriculum? 

o To whet degree do students hove choice over Learning strategies? 

o To what degree do students have choice over materials? 

o Does the curriculum help you better understand the 
company as a whole and how its ports Inter-relate, its 
operations, and how your port relates to the whole? 

o How do you know If you ore eccomplishing your objectives? 
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o How Is language use outside the classroom (on the "shop floor" ) 

connected with language used in the classroom? 

o How ore ESL literacy and language practices Integrated? 

Is real-life language encouraged? or do you mostly fill out workbook 

poges? 

o What knowledge, skills and experiences do you feel ore particularly 
useful for workplace ed. teachers to houe? 
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EUALURTION TERM GROUP INTERUIEW 



PHILOSOPHY, GOALS, MODEL, RPPRORCH 

o Ulhat it your education program s philosophy or 
Instructional approach? 

o Ulhat are its goals? Where Is the curriculum supposed to 
help students end up? 

CURRICULUM 

o Houi was the curriculum created? which partners been 
Inuolued? (Teachers? Education Coordlnotor? Employer? 
Union Rep.? Supervisors? Student/workers? Others? Hf so, 
in uihat ways? Rnd to what degree? 

Sources of Curriculum Goals 

o To what degree does your program use as the context for 
students' learning objectives: 

o Language and/or other basic skills needs at work 

o Issues et work 

o Basic Skills Needs In Worker/students' Lives Outside 
Work (Children's' schoolworfc, writing notes 
to a child's teacher, reading to children, reading 
the Bible, doing Income tax, etc.) 

o Union/Lobor issues 

o Language/grammar needs at work or outtlde work 
o "Life skills" 
o Other? 
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o To what degree do student/workers have choice ouer curriculum 

objectives? 

o Ooes the curriculum or instruction address the student/worker*' 
understanding of the company as a whole and how Its parts 
Inter-relate, Its operations, and how the student/worker's port 
relates to the whole? 

o Whot do you consider the major areas of Innovation In your 
curriculum? 

o Whot do you consider the major problems or limitations of your 
curriculum? 

o What are the curriculum components you want/need most help with? 

IMPACT OF THE CURRICULUM OUTSIDE THE CLASSROOM 

o Is there evidence that what is Included In the curriculum has Impact 
on the company or union: on company or union culture, I.e. practices, 
policies, or priorities? 

o How does the curriculum/classroom respond to learner motivations, 
both extrinsic Chigher wages, bonuses, promotions) and intrinsic (public 
recognition of achievement, appreciation, self-esteem, Increased 
responsibilities)? 

o Does the curriculum or Instruction address tha Issue of Increasing 
student/workers' self confidence? If so, how? 

o How Is language use outside the clossroom (on the "shop floor" ) 
connected with longuoge used In the classroom? 

o How ore teachers oriented to the program? Does the orientation 
include familiarizing them with the work objectives, routines, values 
and terms of the business/industry involved in the partnership? 

o Whot knowledge, skills and experiences are particularly useful for 
workplace ed. teachers to hove? 
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PROGRAM PRODUCTS 

o Are there any program products? Published student uniting? 
Newsletters? Uldeotopes? Community-related projects?) 

INFORM I N6 OTHERS ABOUT THE PR06RRM 

o Horn do you keep people Informed about what Is going on the 
project? 
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TEACHER INTERVIEW PROTOCOL 



TEACHER BACKGROUND 
o What are your major responsibilities? 
Teaching? 

Curriculum development? 

Assessment? 

Others? 

o (Show class schedule.) 

Ooes this accurately reflect the number and kind of dosses you are 
teaching now? (If not, correct.) 

o Are there others who assist you? (aides, volunteers, support staff?) 
If so, who? 

PHILOSOPHY EOALS, MODEL. APPROACH 

o What is your education program's philosophy or instructional 
approach? 

o What are Its goals? Where Is the curriculum supposed to help 
students end up? 

o Whot is its model, especially with regard to: 
o curriculum development 

o Do you view curriculum development as a process 
of helping student/workers (as a group and/or 
individually) articulate their interests, needs, 
goals, and concerns and addressing these? 
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o If so, Is this process described In print? 
(If so, get e copy.) 

o If so, con you describe the process? 
o teaching; 

0 selection nf materials and methods: 

© student inunluamant in daclatan-malrlnafif this it not 

answered by the question "whet Is Its philosophy") 

o lllhet do student/workers hove to do in the program In order to get 
something from It? 

CURRICULUM AND CURRICULUM D EVELOPMENT PROCESS 



o Haue you been (nuolued in creating the curriculum? If so, in what 
ways? And to what degree? 

o Who else is Involued with curriculum development? 
Student/workers? Coordinator? Supervisors? Union? Others? 

o Is the curriculum development process ongoing? 

o Is curriculum development on ongoing process that 
goes bock and forth between the written design, octuol 
implementation, end revision of the design? 

o If so, how much time do you spend eoch week/month 
on this? 

o fire you paid for your curriculum development time? 
Rll of it or some of it? 

Sources of Curriculum coals 

o To what degree does your program use as the context for students' 
learning objectives: 
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0 Language/grammar needs at mark 



o Other basic skills needs at work 

n Itiuet at mork 

o Union/labor lituei/concemt 

n Student/marker emnnmerment 

p Basic Skittl Needs In Worker/students' Uilil 

Outside mark (Children's' schoolwork, writing notes 
to a child's teacher, reading to children, reading 
the Bible, doing Income tax, etc.) 

0 Language/grammar needt nutilde mark 

o Ufejtitllli 

O Indluidual grnmth 

o Family IfSUBt 
0 Cnmmunlty Issues 
o Other? 

o To whet degree do student/workers have choice ouer curriculum 
objectives? 

o If the curriculum is related to the work content, how exactly Is this 
relating done? (e.g. did/does the teecher go to the shop floor end 
collect realla? Other ways?) 

o What skills are addressed In the curriculum: lit toning? speaking? 
reading? writing? grammar? computation? Interpreting skills? life 
skills/survival skills? team-work/co-operation? ability to work 
independently? critical thinking? problem-solving? (Whet does this 
mean In this context? Task analysis? 6lving Instructions orally and In 
writing? Something else?) how to learn about others' cultures? 
management skills? secondary level subject areas? (which ones?) 
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others? 



o Does the curriculum address the Issue of different levels of skills In 
the classroom? If so, hoiu? 

o How is the curriculum orgenlzed? If ESL, Is the curriculum 
grammar-based, skills-based, orgenlzed by life skills or work skills 
competencies? 

o Does the curriculum address the student/workers' understanding of 
the company as a whole and how its parts inter-relate, its operations, 
and how the student/worker's part relates to the whole? 

o What do you consider the mejor erees of Innovotlen In your 
curriculum? 

o What do you consider the mejor problems or limitations of your 
curriculum? 

INSTRUCTIONAL METHODS AND HCTIUITIES 

o What methods, activities, practices, and/or teaching/learning 
strategies do you use In your clesses? 

presentation/lecture? 

discussion? 

drill end practice? 

case studies? 

co-oneratlue actiulties? 

dialog Journals? 
Jazz chants? 

picture stories? 
uidfiAi2 




uncahulary daualnpad from on-site reapnmihllltlei? 
reading comprflhentlnn dariuad from family stories? 

slmulatlon/roie-plngf? 

CRI? 

Modelling skills? 

o (If ESL) are language and literacy practices Integrated In the 
classroom? Describe, 

o (If ESL) is "real life language" encouraged? Describe. 

o Are the methods and activities tailored to the worksite and work 
needs? 

o Are methods and activities tailored to students' life skills needs? 

o To what degree do student/workers have choice over Learning 
strategies/activities? 

o Does instruction address the student/workers' understanding of the 
company as a whole and how its parts Inter-relate, Its operations, and 
how the student/worker's port relates to the whole? 

INSTRUCTIONAL MATERIALS 

o IVhet materials are used? Print and non-print? Pictures? 
Tactile? 

o Where do they come from? Teacher-made? Commercial? Both? 
o How are they evaluated? Who evaluates them? 

Are the materials screened to moke sure they are based 
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on adult subject matter? 

Are materials screened for ethnicity issues? 

Are materials screened for gender issues? 

Are materials screened for other purposes? (What?) 

o To what degree do student/workers have choice ouer materials the 
class uses? 

o To what degree do they have choice ouer materials they, as 
individual learners, use? 



PROGRAM mm ISSUES 

o Do you use LE.P.'s? If so, do they include student learning 
objectives? Are they derived, In part, from the Initial assessment? Do 
they include Information about the student's preferred looming styles? 

o To what degree is the program self-paced, following the objectives 
in on I.E.P., and to what extent group-paced? 

o Are there specific, structured incentlues for student learning 
progress: higher wages, bonuses, promotions, public recognition of 
achievement, appreciation, increased responsibilities, or others) if so, 
what are they? 

o Does the program address the issue of preferred learning styles ( 
collaborative, independent, dependent)? If so, how? 

o Does the curriculum or instruction address the Issue of increasing 
student/workers' self confidence? If so, how? 

o Is your stoff preparation time paid for? 

o Are there regular times set aside for stoff meetings? For what 
purposes? Collaborate to write curriculum? Educational Program 
Design? 
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STUFF ORIENTATION AND STUFF DEUFLOPMENT 

o How were you oriented to the program? 

o Hom are other teachers oriented? 

o Does orientation Include fomlliorfzing teachers with the work 
objectives, personnel, structure, routines, ualues end terms of the 
business/industry (compony and union) Involved In the partnership? 

o What ongoing staff development/training Is there? Is It paid time? 
Is It related to curriculum development? Is It structured? How? Is It 
connected to your evaluation? If so, hom? Does It Include cultural 
sensitlvlty/aivereness? 

o Are there peid staff development opportunities outside the program 
(e.g. stete and national conferences?) If so, give examples. 

o Do you and/or other staff In the program network/coordinate with 
other workplace education programs? Other literacy programs? If so, 
give examples. 

IMPOCT or the curriculum on won*, comprnv, bnd union 

o Is there evidence that what Is included In the curriculum has Impact 
on the compony or union: on compony or union culture, i.e. practices, 
policies, or priorities? If so, describe. 

o How is language use outside the classroom (on the "shop floor") 
connected with longuage used In the classroom? 

Are work-related longuage needs brought (by students, 
supervisors, by you) Into the classroom? 

Ore longuage skills (vocabulary, syntax, 
pronunciation, etc) learned in the classroom used in 
the workplace? 



TEACHERS' PURL I Fl CATIONS 




o What knowledge, skills and experiences era particularly useful for 
workplace ed. teachers to have? 

PROGRAM PRODUCTS 

o Are there any program products? Published student writing? 
Newsletters? Videotapes? Community-related projects?) If so, what? 

INFORMING OTHERS ABOUT THE PBDfiRflM 

a How do you keep people Informed about what is going on the 
project? 



TABLE ft 
SKILLS f 



i.i. say a 



Listening 
Speaking 
Reading 
Writing 

Computation 

Interpreting skills 

Life skills/survival skills 

T earn- uork/co op erat ion 

Ability to uork independently 

Critical thinking and 
Problem solving 

How to leam about others 9 
cultures 

Manag emen t skills 

Secondary level subject areas 
Others 

4 Jo 



x 

X 
X 
X 

x Sometimes drill & practice 
as a supplement 

x Some, depending on the 

company's needs. Measurement 

Not much 

x 

X 
X 

x Included in some activities 
x 

x With S-P.C. to some extent 
especially sequencing 

No. 

Co-operative learning skills 
Peer -teaching 



x 
x 
x 

x (not a priority; journal writing 
newsletter) 

x (as part of language learning — 
in teachers' he ads -not a starting 
point. Addressed in context, 
x (but not a priority) 



no 

x (but not a priority) 

Not in curriculum, but sometimes in 

class it is addressed 

x 

x < Analytical thinking) 



x (This is a goal of the work- 
shops) 

? 



No 

Hetaco^utive skills - nhat kind 
of learners they are; exposing 
workers to different kinds of 
learning activities, so that they 
choose those that they find wor 
best for them > 



ttanpdcn Papers™ Inc./ ILl" " Boston City Hospital 



x x 

x <a little) x 

* x 

* X 

x x 



No No 



X 

No 

x (self reliance) 



x (second level priority) 

x 



x 



x <te some degree. Want to do more) x 
fprxb. solv. mostly with «ath> 

y (tut not recently. Did this x 
with gr-oup learning photography 

^ x (self-nanageoent > 

X ((ED subjects) x (science, research, study skills) 

note taking skills, test taking 

42 ^.ns 
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THE B 

II5TRUCTI0NRL METHODS 
PSD ACTIVITIES 

Prasentat ion/ lecture 

Discussion 

Drill and Practice 

Case studies 

Cooperative activities 

Dialog Journals 



Kennedy/ Qui 



Not ituch 

x 

x Some 
x 

x This is emphasized. 
Plan to use in class 



J«z chants 
Picture stories 
Videos 

U»cab. developed from on-site 
nsponsibi 1 it ies 

fading comprehension derived 
from family stories 

Seoul at ions or role-plays 

Oapuisr — assi sted instruction 

Midelling skills 

Others 

4 jo 
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coming 
up. Currently use "what 
we did today" log. Want to 
use as an assessment tool. 

x One teacher uses these. 

x This is emphasized* 

x Die-casting, "Norma ?ae w 



x 



x 
x 

No. 



Choral work 
Games 

Newspapers - current events 
Morlr mat en a 1 s and tool staples 
fe.g, defective parts) 



South Cove/C.H.C.B. 



x 

x Cincl. the use of flashcards 
to teach promnc i at ion) 



x 



x 



No 

X 

x Language experience charts, 
patient name chart 
Kystery 



Haopden Papers, Inc./ I.L.I. 



Boston City Hospital 



No 

No. (Rarely. Only when there is a 
group. > 

x ( in C.R. I. after students under 
stand the thought behind it) 
x tan an individual basis) 

ft? 

No 



x 

X 

x (practice: role-playing around 
prob. solving. pronouncing vocabulary 
x 

x (role-playing, case studies, 
dissections groups, debates) 
No 



Ito (individual now -tried in first 

year) 

* (a lot) 

k "Dangers in the workplace" video 

k (soae, esp. when a worker *bids* 
in a new area) 

No 



No 

X < budgeting, database, graphics, 
spreadsheets. Not C.B.I. Tools.) 

Cotffuter tools: word processing 



Ho 
No 

x (science — huean systems. "Eyes on 
the Prize" while doing aath work 
x (yes, a lot) 



No 



x (with problea solving issues Paul 
will act out an exaeple) 
Tutoring (Paul does a lot of this 
outside class) one or two field 
trips 
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TABLE C 

1H5TPUCTI0NRL MRTEBIfiLS 



Kennedy/ Quinsigamond 



Materials used 
Commercial? 



Rrint and non-print materials 



Tactile 



How materials are evaluated 



Teacher-made Materials are 
modelled on "English at Work— 
B Toolkit." filso use texts: e.g. 
"ESL for Action, 0 "Speaking t*> at 
Work, " idiom books, Regents 
grammar, and other generic 
trade books 

Field-trips for the teacher in 
the factory, guided by students 

Products company makes, 
and defective parts 
Safety equipment: glasses 
and gloves 

Teachers evaluate curriculum 
units developed by teacher/ 
CLV-riculum developer, to 
extent 



Materials screened to be sure 
that they are based on adult 
subject matter 

Workers choice over materials 
that they as individual 
learners use 



No formal process 



No 
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South Cove Manor /C.H.C.fi. 



Both, but primarily teacher-made 



x (charts, cuisenaire rods, 
drawings, diagrams* 



X (Workers are very much encour- 
aged to experience a variety of 
materials and methods, and to 
choose those which they find to 
work for them. > 



Hsapden Papers, Inc./ I.L.I. 



nfcps, graphs, books 
fV-odbcfcion tickets 

fibth 



*bs- Tape recorder for pronuncia- 
tion. Computers, triangles, 
protractors, videot^se 

Yes. 



Bj teacher on an ad-hoc basic 



Mrs, on an ad hoc basis 



Ves. 



3b 
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Boston City Hospital 



Textbooks, worksheets <e.g. gram 
worksheets), vocab. tests (eedicat 
tereinol.) Elsa fluerbach's text. 



Newspaper and eagazine articles for 
reading and analyzing writing 

health & safety, and rights on the 
job videos 

skeleton, auscular systea eodel 

Paul evaluates. Rdvisory board could 
could play this role. Evaluated for 
adult subj. ftaytter, cultural sen- 
sitivity, readability, clarity and 
understandabilifaj- Not specifical- 
ly screened for gender bias, but 
Paul says he would pick this up. 
Also screens out condescending etrl 
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HTfSKH IN rUI.UJVINM A CHKMIUAIi FMlilflf n< f UN TII.RKT 

1) Find out the name of what you are maJcincrr 
(Look at the left side of the ticket) 

2) Find out hev suh you vill be making. 

(Look for the # of tuba in top section of ticket) 

0) Figure out what site mix task (measured in gallons) you 

iittntl Tin 1.1 in niiiiilittf ii T I.iiIim vim Ml I its mn\ ! nu • 

(i Tub s 25 gallons) 

4) i-ina sue erne sine et miser you neee eo use. 
(Sea "Riser" in top section of ticket.) 

£) Fellow directions HI ORDER. Pay attention to 

highlightod dirootiono so you oomo to thorn I Aok firot 
if you have any questions. 

6) Look at lit ingredient . is it dry or wet? 

7) If it is a dry ingredient. 

a) Go find it. Be sure to read the whole name! 

b) It will usually bo in SO gel lea drums. 

Each ingredient has a different weight, so 50 
y^lluiic uf ijiyrudiunl X will not wuigli Lhu liuu 
ao so gallons of ingredient y. nsxo tho label 
tn find nut hnw many lbs. nf& weight nre In each 
drum of the ingredient you need. 

c) Figure out how many whole drums you need. 

J) IP jrwn ulau ««aoJ t*m*.lU wf uitwlUo*. %li.wuu, j^la -ill 

sees «a measure out this part, ifeu win need «a 
decide which container is big enough to hold 
that amount. 

e) You will need to zero out the scale BEFORE you 

weigh your ingredient, so yr.m n*l the r/rjrrer.H 
net w^lyht. * * 

8) If rt is a set — ingredient . 

a) Go find it. 

b) If it is water, add it by meter. 

o) If it iit; another liquid, mu^cuiu it uciay uiLhui 
the 1 -or. 2_ liter .dipper. 

A) n<» tnm *n r.henk thnt it in right 
temperature. 

«) IX? U»c pjuuJu^Liun LiUveL e»<avs» Lu "WeiyU" Lh«= 

liquid, 

1) Get an empty M dish M 

2) Zero out the scale 

**) Weigh the liquid in lbs. 

438 
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9) Add the next ingredient, following the same methods. 



j.u) rollov mixing/wet out directions in order. ™ 

i ij Hlifn nil lnurnllpiil i linvf- fwrn rtilrlnl 1 u rlulil. nrdm . mid — 

all dino^fcianc Jtan Mining hang— boon f ollo jod, look it s 

thp bottom of tUn pagp tn find r.tm right strain wire to Z 

use, and strain the mis with that screen. — 

12) Turn over the production ticket and fill out the stamp = 
on the back with five things? _ 

a) ¥our name 5 

L) Th* dAt& — 

o) 'Thp amount yon mnrip = 

d) The temperature (measure with thermometer) — 

e) The viscosity 

To uittdsuxtt vl&cubity, take a #2 Zahn Cup — 

Stick it fiusn into tne color — 

Pin 1 it M* a 
Hit your stop watch! 

When it's empty, hit atop vat oh acpinl = 

The number of seconds it takes to empty _ 

is the viscosity. Z 

13) Put a sign on the tank with four things: 

5 

a) Ptudu^liuu £ (liww Mppei. lefl &OlTt*V Of tifikAt) — 

b) Name of color you've made (from lower left 

o) Datu a 

ill) Pin i .till it- II !.-» »ji i i iiij I 1 1 ( ?\ inn t .1 ii.4 uf ±ifiyr.} 15 

14) I^t ffSHPl? f9F 3*>9Ut naif an hour. " 

±5) Look et top of production ticket l.r.i si-h whh-.h Htnnh i ni- — 

waking paper this color is to go on. Call that machine ss 

upinLui a v.J h/ivi i \v, .y&nJ a sample &£ that paper down = 

the chut a to you. — 

16) Luuk at, upper right corner of production ticket for — 
rignt draw down wiro tn ihp ami tflflt. on lor ynn have marip, — 

17) finnri i r ample up tn the Qnali tjr rnntrnl (QD nffira — 

18) CLE AH ALL YOVTR EQUIPMENT = 

4 3 J 
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Appendix D: Termination Study Questionnaires 



Friction Materials, Inc. 
Program Termination Questionnaire 



Background 

Friction Materials, Inc. (FMI) is dosing in September 1991. The following 
questions are intended to get key players in the FMI workplace education 
program to reflect on the role their program played in trying to keep their 
company alive and competitive, how the program did or did not live up to its 
promise and why. 

The first set of open-ended questions ask respondents to directly reflect on the 
goals of the FMI ESI program and how they were or were not implemented. 
The second set of questions ask respondents to reflect on what workers learned 
as a result of participating in the program. 

In its 1986 report, "Workplace Basics: the Skills Employers Want", the American 
Society for Training and Development asserts that the employers it surveyed 
want workers to be competent in seven basic skills groups. The breadth of 
these job related basic skills underscores that workplace education entails far 
more than the traditional three R*s. These categories guide the second set of 
questions for the FMI termination evaluation. Questionnaires are designed for 
managers, supervisor, learning provider, participating worker, and union 
representatives. 

The categories are: 

1 . Foundation - knowing how to learn 

2. Competence - reading writing and computation 

3. Communication - listening and oral c r munication 

4. Adaptability • creative thinking and problem solving 

5. Personal Management - self esteem, goal setting, motivation, and 
personal and career development 

6. Group Effectiveness - interpersonal skills, negotiation and teamwork 

7. Influence - organizational effectiveness and leadership 
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Termination Study Questionnaire for FMI 

(Adapted for managers, supervisors, learning provider representatives/program 
coordinator, participating workers, and union representatives) 

Name: 

Title:_ - 

D3t 8 1 

I ength Of Timo Affiliated with Program 

1.) in retrospect, what were the goals of the ESL program? What were your 
hopes for it? 

Explain 



2.) Did the goals fit with other changes taking place in the company? 

Yes 

No 

Explain 



3.) Were the program goals fully implemented? 

Yes 

No 

Explai n 



Probe: major barriers-- 

• no release time: 

• workers working other jobs; 

• teaching staff ; 

• lack of full management support; 

• lack of understanding of role that education plays in the company's long 
term goal to become a quality-oriented "high production organization"; 

• conflict between fulfilling production quotas and supporting the company's 
education program; 

• inadequate time line to bring about changes wanted by Ecklin; 

• only partial commitment to quality awareness and SPC 



• union role 
•other 



4.) What would you do differently, if you could do your program over again?. 
Please be specific and tell me why you make the suggestion you do. 
Probe: major barriers- 

• no release time; 

• workers working other jobs; 

• teaching staff; 

• lack of full management support; 

• lack of understanding of role that education plays in the company's bng * • 
term goal to become a quality-otientecThigh production organization"; 

• conflict between fulfilling production quotas and supporting the company's 
education program; 

• inadequate time line to bring about changes wanted by Ecklin; 

• only partial commitment to quality awareness and SPC 

• union role 

• other 



Do you think that participation in your workplace education program has 
changed the following for workers at FMi? 

Please circle stayed the same, increased a little, or Increased significantly 



1 . Foundation - knowing how to learn 

• Appreciation for the learning process 

stayed the same increased a little increased significantly 

• Not afraid to ask questions about something I don't know 

stayed the same increased a little increased significantly 

• Know where to go for information to answer my questions (to a book, a person, etc.) 

stayed the same increased a little increased significantly 



2. Competence - reading .writing and computation. 



Math Skills: 

• Do basic addition and subtraction 

stayed the same increased a little increased significantly 

• Do basic multiplication and division 

stayed the same increased a little increased significantly 

• Add and subtract simple fractions 

stayed the same increased a little increased significantly 

• Work with decimals 

stayed the same increased a little increased significantly 

• Figure simple percentages 

stayed the same increased a little increased significantly 

• Rood fl rulsr 

stayed the same increased a little increased significantly 



3. Communication - listening and oral communication 

• Understand directions 

stayed the same increased a little increased significantly 

• Need less time in translation 

stayed the same increased a little increased significantly 

• Listens to coworkers' questions 

stayed the same increased a little increased significantly 

• Ask questions clearly 

stayed the same increased a little increased significantly 

• Give presentations 

stayed the same increased a little increased significantly 



4. Adaptability - creative thinking anu problem solving 

• Dont wait for someone else to tell them what to do. Take initiative. 

stayed the same increased a little increased significantly 

• Think about new/better ways to do things at work 

stayed the same increased a little increased significantly 

• Discover and apply new/better ways to do things at work 

stayed the same increased a little increased significantly 
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5. Personal management - self esteem, goal setting, motivation, and personal and 
career development 

• Value opinions and contributions 

stayed the same increased a little increased significantly 

• Offer opinions to others 

stayed the same increased a little increased significantly 

• Plan ahead at work 

stayed the same increased a little increased significantly 

• Comfortable working alone 

stayed the same increased a little increased significantly 

6. Group effectiveness - interpersonal skills, negotiation and teamwork 

• Work well with co-workers 

stayed the same increased a little increased significantly 

• Developed a sense of team work and team identity 

stayed the same increased a little increased significantly 

• Willing to negotiate personal needs and the needs of group with supervisor 

stayed the same increased a little increased significantly 

• Supportive of others efforts to increase their skills 

stayed the same increased a little increased significantly 

7. Influence - organizational effectiveness and leadership 

• Understand what SPC is 

stayed the same increased a little increased significantly 

• Understand the role that SPC plays in overall company strategy to improve production 
and sales 

stayed the same increased a little increased significantly 

• Understand the relationship between the parts of the work process and the whole 
production process 

stayed the same increased a little increased significantly 

• Understand the long term goals of FMI and the essential role that education plays in 
making those goals a reality 

stayed the same increased a little increased significantly 

8. Other job-specific tasks or skills 

Inspect parts (OR OTHER SKILL APPROPRIATE TO FMI) 

stayed the same increased a little increased significantly 
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Spir-it, Inc. 
Program Termination Questionnaire 



Background 

Spir-it terminated its ESL/ABE program in May 1991. The following questions 
are intended to get key players in the SPIR-IT workplace education program to 
reflect on why the program was terminated, and what could have prevented this 
event. 

The first set of open-ended questions ask respondents to directly reflect on the 
goals of the Spir-it program and how they were or were not implemented. The 
second set of questions ask respondents to reflect on what workers learned as 
a result of participating in the program. 

In its 1988 report, "Workplace Basics: the Skills Employers Warn", the American 
Society for Training and Development asserts that the employers it surveyed 
want workers to be competent in seven basic skills groups. The breadth of 
these job related basic skills underscores that workplace education entails far 
more than the traditional three R's. These categories guide the second set of 
questions for the Spir-it termination evaluation. Questionnaires are designed 
for managers, supervisor, learning provider, and participating worker. 

The categories are: 

1. Foundation - knowing how to team 

2. Competence - reading writing and computation 

3. Communication - listening and oral communication 

4. Adaptability - creative thinking and problem solving 

5. Personal Management - self esteem, goal setting, motivation, and 
personal and career development 

6. Group Effectiveness - interpersonal skills, negotiation and teamwork 

7. Influence - organizational effectiveness and leadership 
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Termination Study Questionnaire for Spir-it, Inc. 

(Adapted for managers, supervisor, teaming provider representative/program 

coordinator, and participating workers) 



Name: 

Title: 

Date : 

Length of Time Affiliated with Program 

1.) In retrospect, what were the goals of the ESL program? What were your 
hopes for it? 

Explain 



3.) Did the goals fit with other changes taking place in the company? 

Yes 

No 

Explain 



3.) Were the program goals fully implemented? 

Yes 

No 

Explain 



Probe: major barriers- 

• no release time; 

• workers working other jobs; 

• teaching staff ; 

• lack of full management support; 

• lack of understanding of role that education plays in the company's long 
term goal to become a quality-oriented"high production organization"; 

• conflict between fulfilling production quotas and supporting the company's 
education program; 

• only partial commitment to quality awareness and SPC 

• other 
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4.) What would you do differently, if you could do your program over again?. 
Please be specific and tell me why you make the suggestion you do. 
Probe: major barriers- 

• no release time; 

• workers working other jobs; 

• teaching staff 

• lack of full management support; 

• lack of understanding of role that education plays in the company's long • * 
term goal to become a quality-one nted"high production organization"; 

• conflict between fulfilling production quotas and supporting the company's 
education program; 

•other 



Do you think that participation in your workplace education program has 
changed the following for workers at Spir-it? 

Please circle stayed the same, Increased a little, or increased significantly 



1. Foundation - knowing how to learn 

• Appreciation for the learning process 

stayed the same increased a iittie increased significantly 

• Not afraid to ask questions about something I dont know 

stayed the same increased a iittie increased significantly 
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• Know where to go for information to answer my questions (to a book, a 
person, etc.) 

stayed the same increased a little increased significantly 

2. Competence - reading .writing and computation (Perception of aggregate 

learning) 

Math Skills: 

• Do basic addition and subtraction 

stayed the same increased a little increased significantly 

• Do basic multiplication and division 

stayed the same increased a little increased significantly 

• Add and subtract simple fractions 

stayed the same increased a little increased significantly 

• Work with decimals 

stayed the same increased a little increased significantly 

• Figure simple percentages 

stayed the same increased a little increased significantly 

• Read a ruler 

stayed the same increased a little increased significantly 

Reading: 

• Read job instructions 

stayed the same increased a little increased significantly 

• Understand what "data" is 

stayed the same increased a little increased significantly 

• Understand what it means for a process to be "in control" or "out of control" 

stayed the same increased a little increased significantly 

• Read graphs 

(Specify the different types of graphs that can be read) 

stayed the same increased a little increased significantly 

• Read charts and grids 

stayed the same increased a little increased significantly 

Writing Skills: 

• Fill out forms/complete a job report 

stayed the same increased a little increased significantly 

• Draw graphs 
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stayed the same increased a little increased significantly 

Write a paragraph 

stayed the same increased a little increased significantly 

Write an essay 

stayed the same increased a little increased significantly 



3. Communication - listening and oral communication 

• Understand directions 

stayed the same increased a little increased significantly 

• Need less time in translation 

stayed the same increased a little increased significantly 

• Listens to coworkers' questions 

stayed the same increased a little increased significantly 

• Ask questions clearly 

stayed the same increased a little increased significantly 

• Give presentations 

stayed the same increased a little increased significantly 



4. Adaptability - creative thinking and problem solving 

* Don't wait for someone else to tell them what to do. Take initiative. 

stayed the same increased a little increased significantly 

* Think about new/better ways to do things at work 

stayed the same increased a little increased significantly 

* Discover and apply new/better ways to do things at work 

stayed the same increased a little increased significantly 



5. Personal management - self esteem, goal setting, motivation, and 
personal and career development 

• Value opinions and contributions 

stayed the same increased a little increased significantly 

• Offer opinions to others 

stayed the same increased a little increased significantly 

• Plan ahead at work 

stayed the same increased a little increased significantly 

• Comfortable working alone 
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stayed the same increased a little increased significantly 



6. Group effectiveness - interpersonal skills, negotiation and teamwork 

• Work well with co-workers 

stayed the same increased a little increased significantly 

• Developed a sense of team work and team identity 

stayed the same increased a little increased significantly 

• Willing to negotiate personal needs and the needs of group with supervisor 

stayed the same increased a little increased significantly 

• Supportive of others efforts to increase their skills 

stayed the same increased a little increased significantly 

7. Influence - organizational effectiveness and leadership 

• Understand what SPC is 

stayed the same increased a little increased significantly 

• Understand the role that SPC plays in overall company strategy to improve 

production and sales 

stayed the same increased a little increased significantly 

• Understand the relationship between the parts of the work process an J the 
whole production process 

stayed the same increased a little increased significantly 

• Understand the long term goals of SPIR-IT and the essential role that 

education plays in making those goals a reality 

stayed the same increased a little increased significantly 

8. Other Job-specific tasks or skills 

Inspect parts (OR OTHER SKILL APPROPRIATE TO SPIR-IT) 

stayed the same increased a little increased significantly 
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